From 7505f775989c47b01b5f5d896c8e050878fb879c Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Tue, 25 Aug 2026 01:07:48 +1000 Subject: [PATCH] Decode AV1 palette blocks --- HEIF_IMPLEMENTATION_PLAN.md | 6 +- THIRD-PARTY-NOTICES.TXT | 142 +++++ .../Av1/Entropy/Av1DefaultDistributions.cs | 75 +++ .../Heif/Av1/Entropy/Av1SymbolDecoder.cs | 123 +++++ .../Av1/Prediction/Av1PredictionDecoder.cs | 20 +- .../Heif/Av1/Tiling/Av1BlockModeInfo.cs | 41 ++ .../Tiling/Av1ParseAboveNeighbor4x4Context.cs | 10 +- .../Tiling/Av1ParseLeftNeighbor4x4Context.cs | 9 +- .../Formats/Heif/Av1/Tiling/Av1TileReader.cs | 520 +++++++++++++++++- 9 files changed, 914 insertions(+), 32 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index ba9add2cb2..8316043492 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -70,6 +70,7 @@ This snapshot pins or classifies the available references and failures; it does | `Av1YuvConverter.ConvertToRgb`, `ConvertFromRgb`, scalar row conversion, chroma reconstruction, and chroma downsampling | H.273 formulas 20-31 and the identity, YCgCo, and non-constant-luminance matrix formulas; AV1 section 6.4.2 chroma sample positions | libavif `src/reformat.c` and `src/colr.c` at `092276ce89098ead06db80975173191e5fee1826`; libaom `aom/aom_image.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar behavioral oracle for full/limited-range conversion at 8, 10, and 12 bits. Decode covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with AV1 chroma sample positioning. Encode covers the same plane layouts, using libavif's actual-edge-count box average for subsampled chroma. Later SIMD paths must preserve this scalar behavior. | | `Av1FrameBuffer` high-bit-depth sample layout and `Av1YuvConverter` 10/12-bit packed-pixel conversion | AV1 section 6.4.1 bit depth and H.273 sample-range scaling | libaom `aom_scale/yv12config.h`, `av1/common/idct.c`, and `av1/common/reconintra.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f`; libavif `src/avif.c` and `src/reformat.c` at `092276ce89098ead06db80975173191e5fee1826` | Establish two-byte native sample storage with sample-unit strides for 10/12-bit reconstruction and use ImageSharp's existing `Rgb48` pixel-operation paths in both directions so packed-pixel staging does not reduce high-bit-depth samples to eight bits. | | `Av1PredictionDecoder`, `Av1HighBitDepthPredictor`, `Av1ChromaFromLumaContext`, `Av1PartitionInfo`, and the scalar DC, directional, Paeth, smooth, filter-intra, and chroma-from-luma predictors | AV1 sections 7.11.2 and 7.11.2.3 intra prediction | libaom `aom_dsp/intrapred.c`, `av1/common/reconintra.c`, `av1/common/av1_common_int.h`, `av1/common/blockd.h`, `av1/common/cfl.c`, and `av1/common/cfl.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Behavioral oracle for luma/chroma mode-neighbor addressing, directional upsampling, Paeth selection, smooth normalization, filter-intra taps, high-bit-depth clipping, chroma-from-luma storage/subsampling, and chroma-from-luma row strides. Existing managed scalar tables and predictors remain the implementation base. The WIP rectangular byte-pipeline smooth digest expectations encode width/height-swapped weights and must be replaced only from an independently generated oracle, not regenerated from this implementation. | +| `Av1TileReader` palette mode/color-map parsing, `Av1SymbolDecoder` palette distributions, `Av1BlockModeInfo` palette state, and `Av1PredictionDecoder` palette reconstruction | AV1 sections 5.11.46, 5.11.49, and 7.11.2 palette prediction | libaom `av1/decoder/decodemv.c`, `av1/decoder/detokenize.c`, `av1/decoder/decoder.h`, `av1/common/pred_common.c`, `av1/common/pred_common.h`, and `av1/common/entropymode.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Reuse the existing tile range decoder and frame-owned mode map while porting the normative palette cache merge, high-bit-depth color deltas, tile-adaptive mode/size/index distributions, diagonal color-map traversal, edge padding, and direct palette-sample reconstruction. This is AV1 still-image compression syntax and does not add retained video reference state or any ISO BMFF surface. | | `Av1FrameInfo`, `Av1TileReader`, and `Av1BlockDecoder` transform/coefficient storage | AV1 section 5.11.39 coefficient syntax and section 7.11.2 reconstruction | libaom `av1/decoder/decodetxb.c` and `av1/decoder/decoder.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve separate luma and chroma transform coefficients at monotonically advancing per-plane offsets within each superblock so reconstruction consumes the same transform-block order produced by tile parsing. | | `Av1InverseQuantizer` and `Av1InverseQuantizationLookup` | AV1 section 7.12.3 inverse quantization | libaom `aom_dsp/aom_dsp_common.h`, `av1/common/quant_common.c`, and `av1/decoder/decodetxb.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Select the per-segment matrix level, alias 64-pixel transform dimensions to their adjusted matrices, retain a flat level-15 matrix, and apply the five-bit inverse-matrix weight scale. The large managed lookup remains a single process-wide table. | | `Av1Inverse2dTransformer` and `Av1InverseTransformerFactory` | AV1 section 7.11.2 inverse transform and reconstruction | libaom `av1/common/av1_inv_txfm1d.c`, `av1/common/av1_inv_txfm2d.c`, and `av1/common/idct.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar transform oracle for coefficient-row traversal, intermediate layout, stage ranges, clipping, and high-bit-depth sample addition. The managed 16-bit overload is also used as a parity oracle for the byte overload. | @@ -116,9 +117,10 @@ This assessment is based on the current source after the upstream ImageSharp mer ### AV1 decoder -- The single-still `Av1Decoder` path now parses tile state before allocating and reconstructing one frame, and it disposes the reconstruction planes after pixel conversion. Reference-frame, `show_existing_frame`, and multi-frame ownership remain incomplete. +- The single-still `Av1Decoder` path now parses tile state before allocating and reconstructing one independently decodable frame, and it disposes the reconstruction planes after pixel conversion. It deliberately does not retain animation/video reference frames or implement `show_existing_frame` playback state. - The reconstruction pipeline disables loop filtering, CDEF, super-resolution, loop restoration, and padding with constant flags. These are normative stages when signaled, not optional quality improvements. -- Loop restoration, palette paths, `show_existing_frame`, reference/CDF state, and other syntax paths contain `NotImplementedException` or equivalent unsupported branches. +- Palette mode now reads the normative luma/chroma mode and size CDFs, neighbor color caches, high-bit-depth color syntax, diagonal color-index maps, clipped-edge padding, and direct sample prediction through the existing reconstruction pipeline. The scalar implementation matches the pinned libaom source, but an independently encoded palette AVIF fixture is still required before this path is independently verified. +- Loop restoration and other normative independently decodable still-image syntax paths still contain `NotImplementedException` or equivalent unsupported branches. Tile-local palette CDF adaptation is now present; the remaining still-image frame-context behavior requires a separate source audit without introducing sequence playback state. - The frame buffer now establishes two-byte native sample storage, logical plane rows, and sample-unit block strides for 10/12-bit frames. The active intra-prediction, inverse-transform, and block-reconstruction path selects native 16-bit samples for 10/12-bit frames and has focused pipeline wiring coverage. Independently encoded high-bit-depth AVIF conformance files are still required; chroma-from-luma storage and the other normative reconstruction stages listed below remain incomplete. - `Av1YuvConverter` now consumes the signaled range, supported H.273 matrix coefficients, subsampling, and chroma sample position for 8, 10, and 12-bit output. Its high-bit-depth decode and encode paths use allocator-backed `Rgb48` rows and the existing `PixelOperations` conversions, avoiding the former eight-bit intermediate. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with libavif-compatible box averaging. Constant-luminance and chromaticity-derived matrices and ICtCp remain incomplete. - The inverse-transform path allocates arrays in a per-transform hot path. diff --git a/THIRD-PARTY-NOTICES.TXT b/THIRD-PARTY-NOTICES.TXT index eaf9f73f07..55dc937022 100644 --- a/THIRD-PARTY-NOTICES.TXT +++ b/THIRD-PARTY-NOTICES.TXT @@ -30,3 +30,145 @@ See the LICENSE file in the CoreFX project root for more information. https://github.com/dotnet/corefx/blob/17300169760c61a90cab8d913636c1058a30a8c1/LICENSE.TXT https://github.com/dotnet/corefx/blob/17300169760c61a90cab8d913636c1058a30a8c1/src/Common/src/CoreLib/System/IO/Stream.cs#L742 https://github.com/dotnet/corefx/blob/17300169760c61a90cab8d913636c1058a30a8c1/src/Common/src/CoreLib/System/IO/Stream.cs#L775 + + +License notice for the AOM AV1 Codec Library +----- + +Parts of the AV1 implementation are adapted from the AOM AV1 Codec Library. +https://aomedia.googlesource.com/aom/ + +Copyright (c) 2016, Alliance for Open Media. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +Alliance for Open Media Patent License 1.0 + +1. License Terms. + +1.1. Patent License. Subject to the terms and conditions of this License, each + Licensor, on behalf of itself and successors in interest and assigns, + grants Licensee a non-sublicensable, perpetual, worldwide, non-exclusive, + no-charge, royalty-free, irrevocable (except as expressly stated in this + License) patent license to its Necessary Claims to make, use, sell, offer + for sale, import or distribute any Implementation. + +1.2. Conditions. + +1.2.1. Availability. As a condition to the grant of rights to Licensee to make, + sell, offer for sale, import or distribute an Implementation under + Section 1.1, Licensee must make its Necessary Claims available under + this License, and must reproduce this License with any Implementation + as follows: + + a. For distribution in source code, by including this License in the + root directory of the source code with its Implementation. + + b. For distribution in any other form (including binary, object form, + and/or hardware description code (e.g., HDL, RTL, Gate Level Netlist, + GDSII, etc.)), by including this License in the documentation, legal + notices, and/or other written materials provided with the + Implementation. + +1.2.2. Additional Conditions. This license is directly from Licensor to + Licensee. Licensee acknowledges as a condition of benefiting from it + that no rights from Licensor are received from suppliers, distributors, + or otherwise in connection with this License. + +1.3. Defensive Termination. If any Licensee, its Affiliates, or its agents + initiates patent litigation or files, maintains, or voluntarily + participates in a lawsuit against another entity or any person asserting + that any Implementation infringes Necessary Claims, any patent licenses + granted under this License directly to the Licensee are immediately + terminated as of the date of the initiation of action unless 1) that suit + was in response to a corresponding suit regarding an Implementation first + brought against an initiating entity, or 2) that suit was brought to + enforce the terms of this License (including intervention in a third-party + action by a Licensee). + +1.4. Disclaimers. The Reference Implementation and Specification are provided + "AS IS" and without warranty. The entire risk as to implementing or + otherwise using the Reference Implementation or Specification is assumed + by the implementer and user. Licensor expressly disclaims any warranties + (express, implied, or otherwise), including implied warranties of + merchantability, non-infringement, fitness for a particular purpose, or + title, related to the material. IN NO EVENT WILL LICENSOR BE LIABLE TO + ANY OTHER PARTY FOR LOST PROFITS OR ANY FORM OF INDIRECT, SPECIAL, + INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF + ACTION OF ANY KIND WITH RESPECT TO THIS LICENSE, WHETHER BASED ON BREACH + OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND WHETHER OR + NOT THE OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +2. Definitions. + +2.1. Affiliate. "Affiliate" means an entity that directly or indirectly + Controls, is Controlled by, or is under common Control of that party. + +2.2. Control. "Control" means direct or indirect control of more than 50% of + the voting power to elect directors of that corporation, or for any other + entity, the power to direct management of such entity. + +2.3. Decoder. "Decoder" means any decoder that conforms fully with all + non-optional portions of the Specification. + +2.4. Encoder. "Encoder" means any encoder that produces a bitstream that can + be decoded by a Decoder only to the extent it produces such a bitstream. + +2.5. Final Deliverable. "Final Deliverable" means the final version of a + deliverable approved by the Alliance for Open Media as a Final + Deliverable. + +2.6. Implementation. "Implementation" means any implementation, including the + Reference Implementation, that is an Encoder and/or a Decoder. An + Implementation also includes components of an Implementation only to the + extent they are used as part of an Implementation. + +2.7. License. "License" means this license. + +2.8. Licensee. "Licensee" means any person or entity who exercises patent + rights granted under this License. + +2.9. Licensor. "Licensor" means (i) any Licensee that makes, sells, offers + for sale, imports or distributes any Implementation, or (ii) a person + or entity that has a licensing obligation to the Implementation as a + result of its membership and/or participation in the Alliance for Open + Media working group that developed the Specification. + +2.10. Necessary Claims. "Necessary Claims" means all claims of patents or + patent applications, (a) that currently or at any time in the future, + are owned or controlled by the Licensor, and (b) (i) would be an + Essential Claim as defined by the W3C Policy as of February 5, 2004 + (https://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential) + as if the Specification was a W3C Recommendation; or (ii) are infringed + by the Reference Implementation. + +2.11. Reference Implementation. "Reference Implementation" means an Encoder + and/or Decoder released by the Alliance for Open Media as a Final + Deliverable. + +2.12. Specification. "Specification" means the specification designated by + the Alliance for Open Media as a Final Deliverable for which this + License was issued. diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs index 45195f166a..f3b9dceeb7 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1DefaultDistributions.cs @@ -113,6 +113,81 @@ internal static class Av1DefaultDistributions /// public static Av1Distribution IntraBlockCopy => new(30531); + /// + /// Gets the luma palette-mode distributions indexed by block-size and neighboring-palette contexts. + /// + public static Av1Distribution[][] PaletteYMode => + [ + [new(31676), new(3419), new(1261)], + [new(31912), new(2859), new(980)], + [new(31823), new(3400), new(781)], + [new(32030), new(3561), new(904)], + [new(32309), new(7337), new(1462)], + [new(32265), new(4015), new(1521)], + [new(32450), new(7946), new(129)] + ]; + + /// + /// Gets the chroma palette-mode distributions indexed by whether luma uses a palette. + /// + public static Av1Distribution[] PaletteUvMode => [new(32461), new(21488)]; + + /// + /// Gets the luma palette-size distributions indexed by block-size context. + /// + public static Av1Distribution[] PaletteYSize => + [ + new(7952, 13000, 18149, 21478, 25527, 29241), + new(7139, 11421, 16195, 19544, 23666, 28073), + new(7788, 12741, 17325, 20500, 24315, 28530), + new(8271, 14064, 18246, 21564, 25071, 28533), + new(12725, 19180, 21863, 24839, 27535, 30120), + new(9711, 14888, 16923, 21052, 25661, 27875), + new(14940, 20797, 21678, 24186, 27033, 28999) + ]; + + /// + /// Gets the chroma palette-size distributions indexed by block-size context. + /// + public static Av1Distribution[] PaletteUvSize => + [ + new(8713, 19979, 27128, 29609, 31331, 32272), + new(5839, 15573, 23581, 26947, 29848, 31700), + new(4426, 11260, 17999, 21483, 25863, 29430), + new(3228, 9464, 14993, 18089, 22523, 27420), + new(3768, 8886, 13091, 17852, 22495, 27207), + new(2464, 8451, 12861, 21632, 25525, 28555), + new(1269, 5435, 10433, 18963, 21700, 25865) + ]; + + /// + /// Gets the luma palette color-index distributions indexed by palette size minus two and spatial context. + /// + public static Av1Distribution[][] PaletteYColorIndex => + [ + [new(28710), new(16384), new(10553), new(27036), new(31603)], + [new(27877, 30490), new(11532, 25697), new(6544, 30234), new(23018, 28072), new(31915, 32385)], + [new(25572, 28046, 30045), new(9478, 21590, 27256), new(7248, 26837, 29824), new(19167, 24486, 28349), new(31400, 31825, 32250)], + [new(24779, 26955, 28576, 30282), new(8669, 20364, 24073, 28093), new(4255, 27565, 29377, 31067), new(19864, 23674, 26716, 29530), new(31646, 31893, 32147, 32426)], + [new(23132, 25407, 26970, 28435, 30073), new(7443, 17242, 20717, 24762, 27982), new(6300, 24862, 26944, 28784, 30671), new(18916, 22895, 25267, 27435, 29652), new(31270, 31550, 31808, 32059, 32353)], + [new(23105, 25199, 26464, 27684, 28931, 30318), new(6950, 15447, 18952, 22681, 25567, 28563), new(7560, 23474, 25490, 27203, 28921, 30708), new(18544, 22373, 24457, 26195, 28119, 30045), new(31198, 31451, 31670, 31882, 32123, 32391)], + [new(21689, 23883, 25163, 26352, 27506, 28827, 30195), new(6892, 15385, 17840, 21606, 24287, 26753, 29204), new(5651, 23182, 25042, 26518, 27982, 29392, 30900), new(19349, 22578, 24418, 25994, 27524, 29031, 30448), new(31028, 31270, 31504, 31705, 31927, 32153, 32392)] + ]; + + /// + /// Gets the chroma palette color-index distributions indexed by palette size minus two and spatial context. + /// + public static Av1Distribution[][] PaletteUvColorIndex => + [ + [new(29089), new(16384), new(8713), new(29257), new(31610)], + [new(25257, 29145), new(12287, 27293), new(7033, 27960), new(20145, 25405), new(30608, 31639)], + [new(24210, 27175, 29903), new(9888, 22386, 27214), new(5901, 26053, 29293), new(18318, 22152, 28333), new(30459, 31136, 31926)], + [new(22980, 25479, 27781, 29986), new(8413, 21408, 24859, 28874), new(2257, 29449, 30594, 31598), new(19189, 21202, 25915, 28620), new(31844, 32044, 32281, 32518)], + [new(22217, 24567, 26637, 28683, 30548), new(7307, 16406, 19636, 24632, 28424), new(4441, 25064, 26879, 28942, 30919), new(17210, 20528, 23319, 26750, 29582), new(30674, 30953, 31396, 31735, 32207)], + [new(21239, 23168, 25044, 26962, 28705, 30506), new(6545, 15012, 18004, 21817, 25503, 28701), new(3448, 26295, 27437, 28704, 30126, 31442), new(15889, 18323, 21704, 24698, 26976, 29690), new(30988, 31204, 31479, 31734, 31983, 32325)], + [new(21442, 23288, 24758, 26246, 27649, 28980, 30563), new(5863, 14933, 17552, 20668, 23683, 26411, 29273), new(3415, 25810, 26877, 27990, 29223, 30394, 31618), new(17965, 20084, 22232, 23974, 26274, 28402, 30390), new(31190, 31329, 31516, 31679, 31825, 32026, 32322)] + ]; + /// /// Gets the partition-type distributions indexed by block-size and neighboring split context. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs index 44726c7a12..19b8c60444 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Entropy/Av1SymbolDecoder.cs @@ -23,6 +23,36 @@ internal ref struct Av1SymbolDecoder /// private readonly Av1Distribution tileIntraBlockCopy; + /// + /// The tile-adaptive luma palette-mode distributions. + /// + private readonly Av1Distribution[][] paletteYMode; + + /// + /// The tile-adaptive chroma palette-mode distributions. + /// + private readonly Av1Distribution[] paletteUvMode; + + /// + /// The tile-adaptive luma palette-size distributions. + /// + private readonly Av1Distribution[] paletteYSize; + + /// + /// The tile-adaptive chroma palette-size distributions. + /// + private readonly Av1Distribution[] paletteUvSize; + + /// + /// The tile-adaptive luma palette color-index distributions. + /// + private readonly Av1Distribution[][] paletteYColorIndex; + + /// + /// The tile-adaptive chroma palette color-index distributions. + /// + private readonly Av1Distribution[][] paletteUvColorIndex; + /// /// The tile-adaptive partition-type distributions. /// @@ -160,6 +190,12 @@ internal ref struct Av1SymbolDecoder // Every tile starts from its own frame-context copy. Sharing these objects would let one image's adaptive // updates change the initial probabilities used to decode the next tile or image. this.tileIntraBlockCopy = Av1DefaultDistributions.IntraBlockCopy.CreateCopy(); + this.paletteYMode = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteYMode); + this.paletteUvMode = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteUvMode); + this.paletteYSize = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteYSize); + this.paletteUvSize = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteUvSize); + this.paletteYColorIndex = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteYColorIndex); + this.paletteUvColorIndex = Av1Distribution.CreateCopy(Av1DefaultDistributions.PaletteUvColorIndex); this.tilePartitionTypes = Av1Distribution.CreateCopy(Av1DefaultDistributions.PartitionTypes); this.keyFrameYMode = Av1Distribution.CreateCopy(Av1DefaultDistributions.KeyFrameYMode); this.uvMode = Av1Distribution.CreateCopy(Av1DefaultDistributions.UvMode); @@ -198,6 +234,93 @@ internal ref struct Av1SymbolDecoder return r.ReadLiteral(bitCount); } + /// + /// Reads an unsigned fixed-width literal from the tile entropy stream. + /// + /// The number of literal bits to read. + /// The decoded literal. + public int ReadLiteral(int bitCount) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadLiteral(bitCount); + } + + /// + /// Reads a uniformly coded value from a non-power-of-two alphabet. + /// + /// The number of possible values. + /// A value in the range from zero through minus one. + public int ReadUniform(int valueCount) + { + ref Av1SymbolReader r = ref this.reader; + int bitCount = Av1Math.Log2(valueCount) + 1; + int threshold = (1 << bitCount) - valueCount; + int value = r.ReadLiteral(bitCount - 1); + if (value < threshold) + { + // The short prefix covers the lower values; only the remaining prefixes consume a final bit. + return value; + } + + return (value << 1) - threshold + r.ReadLiteral(1); + } + + /// + /// Reads whether the current luma block uses palette prediction. + /// + /// The block-area context in the range from zero through six. + /// The number of available above and left luma neighbors that use palettes. + /// when luma palette prediction is selected; otherwise, . + public bool ReadPaletteYMode(int blockSizeContext, int neighborContext) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.paletteYMode[blockSizeContext][neighborContext]) != 0; + } + + /// + /// Reads whether the current chroma block uses palette prediction. + /// + /// A value indicating whether the current block uses a luma palette. + /// when chroma palette prediction is selected; otherwise, . + public bool ReadPaletteUvMode(bool hasLumaPalette) + { + ref Av1SymbolReader r = ref this.reader; + return r.ReadSymbol(this.paletteUvMode[hasLumaPalette ? 1 : 0]) != 0; + } + + /// + /// Reads a luma or chroma palette size. + /// + /// The block-area context in the range from zero through six. + /// The luma or chroma plane class. + /// The palette size in the range from two through eight. + public int ReadPaletteSize(int blockSizeContext, Av1PlaneType planeType) + { + ref Av1SymbolReader r = ref this.reader; + Av1Distribution distribution = planeType == Av1PlaneType.Y + ? this.paletteYSize[blockSizeContext] + : this.paletteUvSize[blockSizeContext]; + + return r.ReadSymbol(distribution) + 2; + } + + /// + /// Reads a palette color-order index from the selected spatial context. + /// + /// The number of colors in the palette. + /// The color-index context derived from decoded neighboring indices. + /// The luma or chroma plane class. + /// The decoded index into the context-specific color order. + public int ReadPaletteColorIndex(int paletteSize, int colorContext, Av1PlaneType planeType) + { + ref Av1SymbolReader r = ref this.reader; + Av1Distribution distribution = planeType == Av1PlaneType.Y + ? this.paletteYColorIndex[paletteSize - 2][colorContext] + : this.paletteUvColorIndex[paletteSize - 2][colorContext]; + + return r.ReadSymbol(distribution); + } + /// /// Reads the frame-local intra-block-copy flag. /// diff --git a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs index f5ee4139c2..cd6321bb36 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Prediction/Av1PredictionDecoder.cs @@ -378,8 +378,24 @@ internal class Av1PredictionDecoder if (usePalette) { - // Palette blocks use their decoded color-index map rather than neighboring-sample - // prediction. The tile parser currently rejects palette syntax before this point. + ReadOnlySpan paletteColors = modeInfo.GetPaletteColors(plane); + ReadOnlySpan colorIndexMap = modeInfo.GetPaletteColorIndexMap(plane); + int paletteStride = partitionInfo.WidthInPixels[(int)plane]; + int mapOffset = ((blockModeInfoRowOffset << Av1Constants.ModeInfoSizeLog2) * paletteStride) + + (blockModeInfoColumnOffset << Av1Constants.ModeInfoSizeLog2); + + // Every transform reconstructs its own window of the block-level palette map. Keeping the map padded to + // the coded block dimensions lets edge transforms use the same addressing rule as interior transforms. + for (int row = 0; row < transformHeight; row++) + { + Span destinationRow = pixelBuffer.Slice(row * pixelBufferStride, transformWidth); + ReadOnlySpan mapRow = colorIndexMap.Slice(mapOffset + (row * paletteStride), transformWidth); + for (int column = 0; column < transformWidth; column++) + { + destinationRow[column] = T.CreateChecked(paletteColors[mapRow[column]]); + } + } + return; } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs index 7bf346dc6e..fe90e274ef 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1BlockModeInfo.cs @@ -15,6 +15,16 @@ internal class Av1BlockModeInfo /// private int[] paletteSize; + /// + /// Stores the decoded palette colors for the Y, U, and V planes. + /// + private readonly ushort[][] paletteColors = [[], [], []]; + + /// + /// Stores the luma and shared chroma palette color-index maps. + /// + private readonly byte[][] paletteColorIndexMaps = [[], []]; + /// /// Initializes a new instance of the class. /// @@ -127,4 +137,35 @@ internal class Av1BlockModeInfo /// The luma palette size. /// The palette size shared by the chroma planes. public void SetPaletteSizes(int ySize, int uvSize) => this.paletteSize = [ySize, uvSize]; + + /// + /// Gets the decoded palette colors for a color plane. + /// + /// The color plane. + /// The palette colors in prediction-index order. + public ReadOnlySpan GetPaletteColors(Av1Plane plane) => this.paletteColors[(int)plane]; + + /// + /// Stores the decoded palette colors for a color plane. + /// + /// The color plane. + /// The palette colors in prediction-index order. + public void SetPaletteColors(Av1Plane plane, ReadOnlySpan colors) + => this.paletteColors[(int)plane] = colors.ToArray(); + + /// + /// Gets the palette color-index map for a color plane. + /// + /// The color plane. + /// The luma map for or the shared chroma map for either chroma plane. + public ReadOnlySpan GetPaletteColorIndexMap(Av1Plane plane) + => this.paletteColorIndexMaps[Math.Min(1, (int)plane)]; + + /// + /// Stores the palette color-index map for a plane class. + /// + /// The luma or shared chroma plane class. + /// The row-major color-index map including coded-block edge padding. + public void SetPaletteColorIndexMap(Av1PlaneType planeType, byte[] colorIndexMap) + => this.paletteColorIndexMaps[(int)planeType] = colorIndexMap; } diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseAboveNeighbor4x4Context.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseAboveNeighbor4x4Context.cs index 91442267f9..829d0e4251 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseAboveNeighbor4x4Context.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseAboveNeighbor4x4Context.cs @@ -7,7 +7,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; /// -/// Stores entropy, palette, partition, and transform contexts for 4-by-4 blocks above the current block. +/// Stores entropy, partition, and transform contexts for 4-by-4 blocks above the current block. /// internal class Av1ParseAboveNeighbor4x4Context { @@ -21,11 +21,6 @@ internal class Av1ParseAboveNeighbor4x4Context /// private readonly int[] aboveSegmentIdPredictionContext; - /// - /// Stores palette base colors for each plane above the current block. - /// - private readonly int[][] abovePaletteColors = new int[Av1Constants.MaxPlanes][]; - /// /// Stores compound-reference group contexts from the preceding 4x4 row. /// @@ -38,13 +33,11 @@ internal class Av1ParseAboveNeighbor4x4Context /// The frame width in 4x4 mode-information columns. public Av1ParseAboveNeighbor4x4Context(int planesCount, int modeInfoColumnCount) { - int wide64x64Count = Av1BlockSize.Block64x64.Get4x4WideCount(); this.AboveTransformWidth = new int[modeInfoColumnCount]; this.AbovePartitionWidth = new int[modeInfoColumnCount]; for (int i = 0; i < planesCount; i++) { this.aboveContext[i] = new int[modeInfoColumnCount]; - this.abovePaletteColors[i] = new int[wide64x64Count * Av1Constants.PaletteMaxSize]; } this.aboveSegmentIdPredictionContext = new int[modeInfoColumnCount]; @@ -83,7 +76,6 @@ internal class Av1ParseAboveNeighbor4x4Context for (int i = 0; i < planeCount; i++) { Array.Fill(this.aboveContext[i], 0, 0, width); - Array.Fill(this.abovePaletteColors[i], 0, 0, width); } Array.Fill(this.aboveSegmentIdPredictionContext, 0, 0, width); diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseLeftNeighbor4x4Context.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseLeftNeighbor4x4Context.cs index 7159036ab8..576e483d85 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseLeftNeighbor4x4Context.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1ParseLeftNeighbor4x4Context.cs @@ -7,7 +7,7 @@ using SixLabors.ImageSharp.Formats.Heif.Av1.Transform; namespace SixLabors.ImageSharp.Formats.Heif.Av1.Tiling; /// -/// Stores entropy, palette, partition, and transform contexts for 4-by-4 blocks left of the current block. +/// Stores entropy, partition, and transform contexts for 4-by-4 blocks left of the current block. /// internal class Av1ParseLeftNeighbor4x4Context { @@ -21,11 +21,6 @@ internal class Av1ParseLeftNeighbor4x4Context /// private readonly int[] leftSegmentIdPredictionContext; - /// - /// Stores palette base colors for each plane left of the current block. - /// - private readonly int[][] leftPaletteColors = new int[Av1Constants.MaxPlanes][]; - /// /// Stores compound-reference group contexts for the current superblock row. /// @@ -43,7 +38,6 @@ internal class Av1ParseLeftNeighbor4x4Context for (int i = 0; i < planesCount; i++) { this.leftContext[i] = new int[superblockModeInfoSize]; - this.leftPaletteColors[i] = new int[superblockModeInfoSize * Av1Constants.PaletteMaxSize]; } this.leftSegmentIdPredictionContext = new int[superblockModeInfoSize]; @@ -76,7 +70,6 @@ internal class Av1ParseLeftNeighbor4x4Context for (int i = 0; i < planeCount; i++) { Array.Fill(this.leftContext[i], 0, 0, blockCount); - Array.Fill(this.leftPaletteColors[i], 0, 0, blockCount); } Array.Fill(this.leftSegmentIdPredictionContext, 0, 0, blockCount); diff --git a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs index a45f20264d..fc22e0f2c9 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Tiling/Av1TileReader.cs @@ -44,6 +44,11 @@ internal class Av1TileReader : IAv1TileReader private static readonly int[][] SkipContexts = [ [1, 2, 2, 2, 3], [1, 4, 4, 4, 5], [1, 4, 4, 4, 5], [1, 4, 4, 4, 5], [1, 4, 4, 4, 6]]; + /// + /// Maps the weighted palette-neighbor score hash to its color-index entropy context. + /// + private static readonly int[] PaletteColorIndexContexts = [-1, -1, 0, -1, -1, 4, 3, 2, 1]; + /// /// Stores the preceding self-guided restoration coefficients for each color plane. /// @@ -55,12 +60,12 @@ internal class Av1TileReader : IAv1TileReader private int[][][] referenceLrWiener = []; /// - /// Tracks entropy, partition, transform, and palette state above the current block. + /// Tracks entropy, partition, and transform state above the current block. /// private readonly Av1ParseAboveNeighbor4x4Context aboveNeighborContext; /// - /// Tracks entropy, partition, transform, and palette state left of the current block. + /// Tracks entropy, partition, and transform state left of the current block. /// private readonly Av1ParseLeftNeighbor4x4Context leftNeighborContext; @@ -440,7 +445,7 @@ internal class Av1TileReader : IAv1TileReader partitionInfo.PopulateModeInfoNeighbors(this.SequenceHeader.ColorConfig); this.ReadModeInfo(ref reader, partitionInfo); - ReadPaletteTokens(ref reader, partitionInfo); + this.ReadPaletteTokens(ref reader, partitionInfo); this.ReadBlockTransformSize(ref reader, modeInfoLocation, partitionInfo, superblockInfo, tileInfo); if (partitionInfo.ModeInfo.Skip) { @@ -1135,18 +1140,54 @@ internal class Av1TileReader : IAv1TileReader /// /// The tile symbol decoder. /// The current coding block. - /// The block selects a nonempty luma or chroma palette. /// Implements AV1 section 5.11.49. - private static void ReadPaletteTokens(ref Av1SymbolDecoder reader, Av1PartitionInfo partitionInfo) + private void ReadPaletteTokens(ref Av1SymbolDecoder reader, Av1PartitionInfo partitionInfo) { - if (partitionInfo.ModeInfo.GetPaletteSize(Av1PlaneType.Y) != 0) + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + if (modeInfo.GetPaletteSize(Av1PlaneType.Y) != 0) { - throw new NotImplementedException(); + GetPaletteMapDimensions( + partitionInfo, + Av1PlaneType.Y, + this.SequenceHeader.ColorConfig, + out int planeWidth, + out int planeHeight, + out int rows, + out int columns); + + byte[] colorIndexMap = DecodePaletteColorMap( + ref reader, + modeInfo.GetPaletteSize(Av1PlaneType.Y), + Av1PlaneType.Y, + planeWidth, + planeHeight, + rows, + columns); + + modeInfo.SetPaletteColorIndexMap(Av1PlaneType.Y, colorIndexMap); } - if (partitionInfo.ModeInfo.GetPaletteSize(Av1PlaneType.Uv) != 0) + if (modeInfo.GetPaletteSize(Av1PlaneType.Uv) != 0) { - throw new NotImplementedException(); + GetPaletteMapDimensions( + partitionInfo, + Av1PlaneType.Uv, + this.SequenceHeader.ColorConfig, + out int planeWidth, + out int planeHeight, + out int rows, + out int columns); + + byte[] colorIndexMap = DecodePaletteColorMap( + ref reader, + modeInfo.GetPaletteSize(Av1PlaneType.Uv), + Av1PlaneType.Uv, + planeWidth, + planeHeight, + rows, + columns); + + modeInfo.SetPaletteColorIndexMap(Av1PlaneType.Uv, colorIndexMap); } } @@ -1293,10 +1334,467 @@ internal class Av1TileReader : IAv1TileReader /// /// The tile symbol decoder. /// The current coding block. - /// Palette-mode syntax is not implemented. /// Implements AV1 section 5.11.46. private void PaletteModeInfo(ref Av1SymbolDecoder reader, Av1PartitionInfo partitionInfo) - => throw new NotImplementedException(); + { + Av1BlockModeInfo modeInfo = partitionInfo.ModeInfo; + Av1BlockSize blockSize = modeInfo.BlockSize; + + // The palette block-size context is the base-two block-area difference from an 8-by-8 block. + int blockSizeContext = Av1Math.Log2(blockSize.GetWidth() * blockSize.GetHeight()) - 6; + int yPaletteSize = 0; + int uvPaletteSize = 0; + int bitDepth = this.SequenceHeader.ColorConfig.BitDepth.GetBitCount(); + if (modeInfo.YMode == Av1PredictionMode.DC) + { + int neighborContext = 0; + if (partitionInfo.AboveModeInfo is not null && partitionInfo.AboveModeInfo.GetPaletteSize(Av1PlaneType.Y) != 0) + { + neighborContext++; + } + + if (partitionInfo.LeftModeInfo is not null && partitionInfo.LeftModeInfo.GetPaletteSize(Av1PlaneType.Y) != 0) + { + neighborContext++; + } + + if (reader.ReadPaletteYMode(blockSizeContext, neighborContext)) + { + yPaletteSize = reader.ReadPaletteSize(blockSizeContext, Av1PlaneType.Y); + Span yColors = stackalloc ushort[Av1Constants.PaletteMaxSize]; + ReadPaletteColorsY(ref reader, partitionInfo, yPaletteSize, bitDepth, yColors); + modeInfo.SetPaletteColors(Av1Plane.Y, yColors[..yPaletteSize]); + } + } + + if (this.SequenceHeader.ColorConfig.PlaneCount > 1 && + modeInfo.UvMode == Av1PredictionMode.DC && + partitionInfo.IsChroma && + reader.ReadPaletteUvMode(yPaletteSize != 0)) + { + uvPaletteSize = reader.ReadPaletteSize(blockSizeContext, Av1PlaneType.Uv); + Span uColors = stackalloc ushort[Av1Constants.PaletteMaxSize]; + Span vColors = stackalloc ushort[Av1Constants.PaletteMaxSize]; + ReadPaletteColorsUv(ref reader, partitionInfo, uvPaletteSize, bitDepth, uColors, vColors); + modeInfo.SetPaletteColors(Av1Plane.U, uColors[..uvPaletteSize]); + modeInfo.SetPaletteColors(Av1Plane.V, vColors[..uvPaletteSize]); + } + + modeInfo.SetPaletteSizes(yPaletteSize, uvPaletteSize); + } + + /// + /// Reads the sorted luma palette colors, including selections from neighboring palette caches. + /// + /// The tile symbol decoder. + /// The current coding block and its palette neighbors. + /// The number of luma palette colors. + /// The number of bits in each color sample. + /// The destination palette-color buffer. + private static void ReadPaletteColorsY( + ref Av1SymbolDecoder reader, + Av1PartitionInfo partitionInfo, + int paletteSize, + int bitDepth, + scoped Span colors) + { + Span colorCache = stackalloc ushort[Av1Constants.PaletteMaxSize * 2]; + Span cachedColors = stackalloc ushort[Av1Constants.PaletteMaxSize]; + int cacheSize = GetPaletteCache(partitionInfo, Av1Plane.Y, colorCache); + int colorIndex = 0; + for (int i = 0; i < cacheSize && colorIndex < paletteSize; i++) + { + if (reader.ReadLiteral(1) != 0) + { + cachedColors[colorIndex++] = colorCache[i]; + } + } + + if (colorIndex == paletteSize) + { + cachedColors[..paletteSize].CopyTo(colors); + return; + } + + int cachedColorCount = colorIndex; + colors[colorIndex++] = (ushort)reader.ReadLiteral(bitDepth); + if (colorIndex < paletteSize) + { + int bits = bitDepth - 3 + reader.ReadLiteral(2); + int maximumColor = (1 << bitDepth) - 1; + int range = maximumColor - colors[colorIndex - 1]; + for (; colorIndex < paletteSize; colorIndex++) + { + int delta = reader.ReadLiteral(bits) + 1; + colors[colorIndex] = (ushort)Av1Math.Clip3(0, maximumColor, colors[colorIndex - 1] + delta); + range -= colors[colorIndex] - colors[colorIndex - 1]; + bits = Math.Min(bits, (int)Av1Math.CeilLog2((uint)range)); + } + } + + MergePaletteColors(colors, cachedColors, paletteSize, cachedColorCount); + } + + /// + /// Reads the U and V palette colors, including neighboring U colors and optional V delta coding. + /// + /// The tile symbol decoder. + /// The current coding block and its palette neighbors. + /// The number of chroma palette colors. + /// The number of bits in each color sample. + /// The destination U palette-color buffer. + /// The destination V palette-color buffer. + private static void ReadPaletteColorsUv( + ref Av1SymbolDecoder reader, + Av1PartitionInfo partitionInfo, + int paletteSize, + int bitDepth, + scoped Span uColors, + scoped Span vColors) + { + Span colorCache = stackalloc ushort[Av1Constants.PaletteMaxSize * 2]; + Span cachedColors = stackalloc ushort[Av1Constants.PaletteMaxSize]; + int cacheSize = GetPaletteCache(partitionInfo, Av1Plane.U, colorCache); + int colorIndex = 0; + for (int i = 0; i < cacheSize && colorIndex < paletteSize; i++) + { + if (reader.ReadLiteral(1) != 0) + { + cachedColors[colorIndex++] = colorCache[i]; + } + } + + if (colorIndex < paletteSize) + { + int cachedColorCount = colorIndex; + uColors[colorIndex++] = (ushort)reader.ReadLiteral(bitDepth); + if (colorIndex < paletteSize) + { + int bits = bitDepth - 3 + reader.ReadLiteral(2); + int maximumColor = (1 << bitDepth) - 1; + int range = (1 << bitDepth) - uColors[colorIndex - 1]; + for (; colorIndex < paletteSize; colorIndex++) + { + int delta = reader.ReadLiteral(bits); + uColors[colorIndex] = (ushort)Av1Math.Clip3(0, maximumColor, uColors[colorIndex - 1] + delta); + range -= uColors[colorIndex] - uColors[colorIndex - 1]; + bits = Math.Min(bits, (int)Av1Math.CeilLog2((uint)range)); + } + } + + MergePaletteColors(uColors, cachedColors, paletteSize, cachedColorCount); + } + else + { + cachedColors[..paletteSize].CopyTo(uColors); + } + + if (reader.ReadLiteral(1) != 0) + { + // V deltas wrap in the unsigned sample domain so complementary chroma colors remain compact. + int bits = bitDepth - 4 + reader.ReadLiteral(2); + int maximumColorPlusOne = 1 << bitDepth; + vColors[0] = (ushort)reader.ReadLiteral(bitDepth); + for (int i = 1; i < paletteSize; i++) + { + int delta = reader.ReadLiteral(bits); + if (delta != 0 && reader.ReadLiteral(1) != 0) + { + delta = -delta; + } + + int value = vColors[i - 1] + delta; + if (value < 0) + { + value += maximumColorPlusOne; + } + + if (value >= maximumColorPlusOne) + { + value -= maximumColorPlusOne; + } + + vColors[i] = (ushort)value; + } + } + else + { + for (int i = 0; i < paletteSize; i++) + { + vColors[i] = (ushort)reader.ReadLiteral(bitDepth); + } + } + } + + /// + /// Builds the sorted unique palette cache from the available above and left block palettes. + /// + /// The current coding block and its decoded neighbors. + /// The luma or U plane whose sorted base colors form the cache. + /// The destination cache, which can hold both neighboring palettes. + /// The number of colors written to . + private static int GetPaletteCache(Av1PartitionInfo partitionInfo, Av1Plane plane, Span cache) + { + // AV1 deliberately excludes the block above at a 64-by-64 superblock-row boundary. + int minimumSuperblockHeight = Av1BlockSize.Block64x64.Get4x4HighCount(); + Av1BlockModeInfo? aboveModeInfo = partitionInfo.RowIndex % minimumSuperblockHeight == 0 + ? null + : partitionInfo.AboveModeInfo; + Av1BlockModeInfo? leftModeInfo = partitionInfo.LeftModeInfo; + + int abovePaletteSize = aboveModeInfo?.GetPaletteSize(plane) ?? 0; + int leftPaletteSize = leftModeInfo?.GetPaletteSize(plane) ?? 0; + ReadOnlySpan aboveColors = aboveModeInfo is null ? [] : aboveModeInfo.GetPaletteColors(plane); + ReadOnlySpan leftColors = leftModeInfo is null ? [] : leftModeInfo.GetPaletteColors(plane); + int aboveIndex = 0; + int leftIndex = 0; + int count = 0; + while (aboveIndex < abovePaletteSize && leftIndex < leftPaletteSize) + { + ushort aboveColor = aboveColors[aboveIndex]; + ushort leftColor = leftColors[leftIndex]; + if (leftColor < aboveColor) + { + AddPaletteCacheColor(cache, ref count, leftColor); + leftIndex++; + } + else + { + AddPaletteCacheColor(cache, ref count, aboveColor); + aboveIndex++; + if (leftColor == aboveColor) + { + leftIndex++; + } + } + } + + while (aboveIndex < abovePaletteSize) + { + AddPaletteCacheColor(cache, ref count, aboveColors[aboveIndex++]); + } + + while (leftIndex < leftPaletteSize) + { + AddPaletteCacheColor(cache, ref count, leftColors[leftIndex++]); + } + + return count; + } + + /// + /// Appends a palette cache color unless it duplicates the preceding sorted value. + /// + /// The sorted cache being populated. + /// The number of colors currently stored. + /// The next sorted color. + private static void AddPaletteCacheColor(Span cache, ref int count, ushort color) + { + if (count == 0 || cache[count - 1] != color) + { + cache[count++] = color; + } + } + + /// + /// Merges selected cached colors with the sorted transmitted colors in one prediction-order palette. + /// + /// The transmitted colors beginning at and the merged output. + /// The selected cached colors in ascending order. + /// The total palette size. + /// The number of selected cached colors. + private static void MergePaletteColors(Span colors, ReadOnlySpan cachedColors, int paletteSize, int cachedColorCount) + { + if (cachedColorCount == 0) + { + return; + } + + int cacheIndex = 0; + int transmittedIndex = cachedColorCount; + for (int i = 0; i < paletteSize; i++) + { + if (cacheIndex < cachedColorCount && + (transmittedIndex >= paletteSize || cachedColors[cacheIndex] <= colors[transmittedIndex])) + { + colors[i] = cachedColors[cacheIndex++]; + } + else + { + colors[i] = colors[transmittedIndex++]; + } + } + } + + /// + /// Computes the padded plane dimensions and the portion that lies inside the coded image. + /// + /// The current coding block and frame-edge distances. + /// The luma or shared chroma plane class. + /// The sequence chroma-subsampling configuration. + /// The padded plane-block width in samples. + /// The padded plane-block height in samples. + /// The number of plane-block rows inside the coded image. + /// The number of plane-block columns inside the coded image. + private static void GetPaletteMapDimensions( + Av1PartitionInfo partitionInfo, + Av1PlaneType planeType, + ObuColorConfig colorConfig, + out int planeWidth, + out int planeHeight, + out int rows, + out int columns) + { + int subX = planeType == Av1PlaneType.Uv && colorConfig.SubSamplingX ? 1 : 0; + int subY = planeType == Av1PlaneType.Uv && colorConfig.SubSamplingY ? 1 : 0; + int blockWidth = partitionInfo.ModeInfo.BlockSize.GetWidth(); + int blockHeight = partitionInfo.ModeInfo.BlockSize.GetHeight(); + int columnsInsideImage = partitionInfo.ModeBlockToRightEdge >= 0 + ? blockWidth + : blockWidth + (partitionInfo.ModeBlockToRightEdge >> 3); + int rowsInsideImage = partitionInfo.ModeBlockToBottomEdge >= 0 + ? blockHeight + : blockHeight + (partitionInfo.ModeBlockToBottomEdge >> 3); + + planeWidth = blockWidth >> subX; + planeHeight = blockHeight >> subY; + columns = columnsInsideImage >> subX; + rows = rowsInsideImage >> subY; + } + + /// + /// Decodes a padded palette color-index map in AV1 diagonal wavefront order. + /// + /// The tile symbol decoder. + /// The number of palette colors. + /// The luma or shared chroma plane class. + /// The padded plane-block width. + /// The padded plane-block height. + /// The number of rows inside the coded image. + /// The number of columns inside the coded image. + /// The decoded row-major color-index map. + private static byte[] DecodePaletteColorMap( + ref Av1SymbolDecoder reader, + int paletteSize, + Av1PlaneType planeType, + int planeWidth, + int planeHeight, + int rows, + int columns) + { + byte[] colorIndexMap = new byte[planeWidth * planeHeight]; + colorIndexMap[0] = (byte)reader.ReadUniform(paletteSize); + Span colorOrder = stackalloc byte[Av1Constants.PaletteMaxSize]; + for (int diagonal = 1; diagonal < rows + columns - 1; diagonal++) + { + int firstColumn = Math.Min(diagonal, columns - 1); + int lastColumn = Math.Max(0, diagonal - rows + 1); + for (int column = firstColumn; column >= lastColumn; column--) + { + int row = diagonal - column; + int colorContext = GetPaletteColorIndexContext( + colorIndexMap, + planeWidth, + row, + column, + paletteSize, + colorOrder); + + int colorOrderIndex = reader.ReadPaletteColorIndex(paletteSize, colorContext, planeType); + colorIndexMap[(row * planeWidth) + column] = colorOrder[colorOrderIndex]; + } + } + + if (columns < planeWidth) + { + // Blocks clipped by the right image edge repeat their final coded column into the padded block area. + for (int row = 0; row < rows; row++) + { + int rowOffset = row * planeWidth; + colorIndexMap.AsSpan(rowOffset + columns, planeWidth - columns) + .Fill(colorIndexMap[rowOffset + columns - 1]); + } + } + + // Blocks clipped by the bottom image edge repeat their final coded row for later transform reconstruction. + ReadOnlySpan finalRow = colorIndexMap.AsSpan((rows - 1) * planeWidth, planeWidth); + for (int row = rows; row < planeHeight; row++) + { + finalRow.CopyTo(colorIndexMap.AsSpan(row * planeWidth, planeWidth)); + } + + return colorIndexMap; + } + + /// + /// Derives the palette color order and entropy context from the left, upper-left, and above indices. + /// + /// The partially decoded color-index map. + /// The map row stride. + /// The current map row. + /// The current map column. + /// The number of palette colors. + /// The destination color order for the current context. + /// The color-index entropy context in the range from zero through four. + private static int GetPaletteColorIndexContext( + ReadOnlySpan colorIndexMap, + int stride, + int row, + int column, + int paletteSize, + Span colorOrder) + { + Span neighborColors = stackalloc int[3]; + neighborColors[0] = column > 0 ? colorIndexMap[(row * stride) + column - 1] : -1; + neighborColors[1] = column > 0 && row > 0 ? colorIndexMap[((row - 1) * stride) + column - 1] : -1; + neighborColors[2] = row > 0 ? colorIndexMap[((row - 1) * stride) + column] : -1; + + Span scores = stackalloc int[Av1Constants.PaletteMaxSize]; + ReadOnlySpan neighborWeights = [2, 1, 2]; + for (int i = 0; i < neighborColors.Length; i++) + { + if (neighborColors[i] >= 0) + { + scores[neighborColors[i]] += neighborWeights[i]; + } + } + + for (int i = 0; i < colorOrder.Length; i++) + { + colorOrder[i] = (byte)i; + } + + // Stable descending score order keeps lower palette indices ahead when neighboring scores tie. + for (int i = 0; i < 3; i++) + { + int maximumScore = scores[i]; + int maximumIndex = i; + for (int j = i + 1; j < paletteSize; j++) + { + if (scores[j] > maximumScore) + { + maximumScore = scores[j]; + maximumIndex = j; + } + } + + if (maximumIndex != i) + { + byte maximumColor = colorOrder[maximumIndex]; + for (int j = maximumIndex; j > i; j--) + { + scores[j] = scores[j - 1]; + colorOrder[j] = colorOrder[j - 1]; + } + + scores[i] = maximumScore; + colorOrder[i] = maximumColor; + } + } + + int contextHash = scores[0] + (2 * scores[1]) + (2 * scores[2]); + return PaletteColorIndexContexts[contextHash]; + } /// /// Reads the joint signs and nonzero alpha magnitudes for chroma-from-luma prediction.