From 12085a35298240b7908f8440c7494b77ab4cc090 Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Tue, 25 Aug 2026 00:37:46 +1000 Subject: [PATCH] Preserve AV1 high bit depth color precision --- HEIF_IMPLEMENTATION_PLAN.md | 4 +- .../Formats/Heif/Av1/Av1YuvConverter.cs | 189 ++++++++++++++---- 2 files changed, 151 insertions(+), 42 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 63f611f19a..aa8ba74c13 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -68,7 +68,7 @@ This snapshot pins or classifies the available references and failures; it does | Managed implementation | Normative behavior | Reviewed implementation reference | Use | | --- | --- | --- | --- | | `Av1YuvConverter.ConvertToRgb`, `ConvertFromRgb`, scalar row conversion, chroma reconstruction, and chroma downsampling | H.273 formulas 20-31 and the identity, YCgCo, and non-constant-luminance matrix formulas; AV1 section 6.4.2 chroma sample positions | libavif `src/reformat.c` and `src/colr.c` at `092276ce89098ead06db80975173191e5fee1826`; libaom `aom/aom_image.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar behavioral oracle for full/limited-range conversion at 8, 10, and 12 bits. Decode covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with AV1 chroma sample positioning. Encode covers the same plane layouts, using libavif's actual-edge-count box average for subsampled chroma. Later SIMD paths must preserve this scalar behavior. | -| `Av1FrameBuffer` high-bit-depth sample layout and `Av1YuvConverter` 10/12-bit output conversion | AV1 section 6.4.1 bit depth and H.273 sample-range scaling | libaom `aom_scale/yv12config.h`, `av1/common/idct.c`, and `av1/common/reconintra.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f`; libavif `src/avif.c` and `src/reformat.c` at `092276ce89098ead06db80975173191e5fee1826` | Establish two-byte native sample storage with sample-unit strides for 10/12-bit reconstruction and use the same scalar color model at every supported bit depth. | +| `Av1FrameBuffer` high-bit-depth sample layout and `Av1YuvConverter` 10/12-bit packed-pixel conversion | AV1 section 6.4.1 bit depth and H.273 sample-range scaling | libaom `aom_scale/yv12config.h`, `av1/common/idct.c`, and `av1/common/reconintra.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f`; libavif `src/avif.c` and `src/reformat.c` at `092276ce89098ead06db80975173191e5fee1826` | Establish two-byte native sample storage with sample-unit strides for 10/12-bit reconstruction and use ImageSharp's existing `Rgb48` pixel-operation paths in both directions so packed-pixel staging does not reduce high-bit-depth samples to eight bits. | | `Av1PredictionDecoder`, `Av1HighBitDepthPredictor`, `Av1ChromaFromLumaContext`, `Av1PartitionInfo`, and the scalar DC, directional, Paeth, smooth, filter-intra, and chroma-from-luma predictors | AV1 sections 7.11.2 and 7.11.2.3 intra prediction | libaom `aom_dsp/intrapred.c`, `av1/common/reconintra.c`, `av1/common/av1_common_int.h`, `av1/common/blockd.h`, `av1/common/cfl.c`, and `av1/common/cfl.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Behavioral oracle for luma/chroma mode-neighbor addressing, directional upsampling, Paeth selection, smooth normalization, filter-intra taps, high-bit-depth clipping, chroma-from-luma storage/subsampling, and chroma-from-luma row strides. Existing managed scalar tables and predictors remain the implementation base. The WIP rectangular byte-pipeline smooth digest expectations encode width/height-swapped weights and must be replaced only from an independently generated oracle, not regenerated from this implementation. | | `Av1FrameInfo`, `Av1TileReader`, and `Av1BlockDecoder` transform/coefficient storage | AV1 section 5.11.39 coefficient syntax and section 7.11.2 reconstruction | libaom `av1/decoder/decodetxb.c` and `av1/decoder/decoder.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve separate luma and chroma transform coefficients at monotonically advancing per-plane offsets within each superblock so reconstruction consumes the same transform-block order produced by tile parsing. | | `Av1InverseQuantizer` and `Av1InverseQuantizationLookup` | AV1 section 7.12.3 inverse quantization | libaom `aom_dsp/aom_dsp_common.h`, `av1/common/quant_common.c`, and `av1/decoder/decodetxb.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Select the per-segment matrix level, alias 64-pixel transform dimensions to their adjusted matrices, retain a flat level-15 matrix, and apply the five-bit inverse-matrix weight scale. The large managed lookup remains a single process-wide table. | @@ -118,7 +118,7 @@ This assessment is based on the current source after the upstream ImageSharp mer - The reconstruction pipeline disables loop filtering, CDEF, super-resolution, loop restoration, and padding with constant flags. These are normative stages when signaled, not optional quality improvements. - Loop restoration, palette paths, `show_existing_frame`, reference/CDF state, and other syntax paths contain `NotImplementedException` or equivalent unsupported branches. - The frame buffer now establishes two-byte native sample storage, logical plane rows, and sample-unit block strides for 10/12-bit frames. The active intra-prediction, inverse-transform, and block-reconstruction path selects native 16-bit samples for 10/12-bit frames and has focused pipeline wiring coverage. Independently encoded high-bit-depth AVIF conformance files are still required; chroma-from-luma storage and the other normative reconstruction stages listed below remain incomplete. -- `Av1YuvConverter` now consumes the signaled range, supported H.273 matrix coefficients, subsampling, and chroma sample position for 8, 10, and 12-bit output. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with allocator-backed RGB rows and libavif-compatible box averaging. Constant-luminance and chromaticity-derived matrices and ICtCp remain incomplete. +- `Av1YuvConverter` now consumes the signaled range, supported H.273 matrix coefficients, subsampling, and chroma sample position for 8, 10, and 12-bit output. Its high-bit-depth decode and encode paths use allocator-backed `Rgb48` rows and the existing `PixelOperations` conversions, avoiding the former eight-bit intermediate. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with libavif-compatible box averaging. Constant-luminance and chromaticity-derived matrices and ICtCp remain incomplete. - The inverse-transform path allocates arrays in a per-transform hot path. - No usable end-to-end AV1 SIMD path was found. The most visible 4x4 forward-transform SIMD call is commented out, while the production prediction, transform, filter, and output paths are predominantly scalar. diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs b/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs index 46db259537..3af88f4fe6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs @@ -20,6 +20,11 @@ internal static class Av1YuvConverter /// private const float ByteMaximum = byte.MaxValue; + /// + /// The largest value represented by a 16-bit packed RGB component. + /// + private const float UShortMaximum = ushort.MaxValue; + /// /// Identifies the matrix operation used between encoded planes and RGB components. /// @@ -69,8 +74,19 @@ internal static class Av1YuvConverter int subY = frameBuffer.ColorConfig.SubSamplingY ? 1 : 0; Buffer2DRegion uPlane = isMonochrome ? default : frameBuffer.DeriveBlockPointer(Av1Plane.U, subX, subY); Buffer2DRegion vPlane = isMonochrome ? default : frameBuffer.DeriveBlockPointer(Av1Plane.V, subX, subY); - using IMemoryOwner rowOwner = configuration.MemoryAllocator.Allocate(image.Width); - Span rgbRow = rowOwner.GetSpan()[..image.Width]; + bool isEightBit = frameBuffer.BitDepth == Av1BitDepth.EightBit; + using IMemoryOwner? rowOwner = isEightBit + ? configuration.MemoryAllocator.Allocate(image.Width) + : null; + + using IMemoryOwner? highBitDepthRowOwner = isEightBit + ? null + : configuration.MemoryAllocator.Allocate(image.Width); + + Span rgbRow = rowOwner is null ? Span.Empty : rowOwner.GetSpan()[..image.Width]; + Span highBitDepthRgbRow = highBitDepthRowOwner is null + ? Span.Empty + : highBitDepthRowOwner.GetSpan()[..image.Width]; for (int y = 0; y < image.Height; y++) { @@ -89,7 +105,7 @@ internal static class Av1YuvConverter out y1Weight); } - if (frameBuffer.BitDepth == Av1BitDepth.EightBit) + if (isEightBit) { ConvertYuvToRgbRow( yPlane.DangerousGetRowSpan(y), @@ -115,6 +131,8 @@ internal static class Av1YuvConverter } else { + // Staging high-bit-depth samples through Rgb48 preserves their precision while still using the + // optimized packed-pixel conversion paths shared by the rest of ImageSharp. ConvertYuvToRgbRow( frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, y, 0, 0), isMonochrome ? default : frameBuffer.GetHighBitDepthRowSpan(Av1Plane.U, y0, subX, subY), @@ -122,7 +140,7 @@ internal static class Av1YuvConverter isMonochrome ? default : frameBuffer.GetHighBitDepthRowSpan(Av1Plane.V, y0, subX, subY), isMonochrome ? default : frameBuffer.GetHighBitDepthRowSpan(Av1Plane.V, y1, subX, subY), y1Weight, - rgbRow, + highBitDepthRgbRow, isMonochrome, subX, subY, @@ -138,10 +156,20 @@ internal static class Av1YuvConverter sampleMaximum); } - PixelOperations.Instance.FromRgb24( - configuration, - rgbRow, - image.PixelBuffer.DangerousGetRowSpan(y)); + if (isEightBit) + { + PixelOperations.Instance.FromRgb24( + configuration, + rgbRow, + image.PixelBuffer.DangerousGetRowSpan(y)); + } + else + { + PixelOperations.Instance.FromRgb48( + configuration, + highBitDepthRgbRow, + image.PixelBuffer.DangerousGetRowSpan(y)); + } } } @@ -174,29 +202,68 @@ internal static class Av1YuvConverter Buffer2DRegion uPlane = isMonochrome ? default : frameBuffer.DeriveBlockPointer(Av1Plane.U, subX, subY); Buffer2DRegion vPlane = isMonochrome ? default : frameBuffer.DeriveBlockPointer(Av1Plane.V, subX, subY); int sourceRowsPerIteration = !isMonochrome && subY != 0 ? 2 : 1; - using IMemoryOwner rowOwner = configuration.MemoryAllocator.Allocate(image.Width * sourceRowsPerIteration); - Span rgbRow0 = rowOwner.GetSpan()[..image.Width]; - Span rgbRow1 = sourceRowsPerIteration == 2 + bool isEightBit = frameBuffer.BitDepth == Av1BitDepth.EightBit; + int rowBufferLength = image.Width * sourceRowsPerIteration; + using IMemoryOwner? rowOwner = isEightBit + ? configuration.MemoryAllocator.Allocate(rowBufferLength) + : null; + + using IMemoryOwner? highBitDepthRowOwner = isEightBit + ? null + : configuration.MemoryAllocator.Allocate(rowBufferLength); + + Span rgbRow0 = rowOwner is null ? Span.Empty : rowOwner.GetSpan()[..image.Width]; + Span rgbRow1 = sourceRowsPerIteration == 2 && rowOwner is not null ? rowOwner.GetSpan().Slice(image.Width, image.Width) : Span.Empty; + Span highBitDepthRgbRow0 = highBitDepthRowOwner is null + ? Span.Empty + : highBitDepthRowOwner.GetSpan()[..image.Width]; + + Span highBitDepthRgbRow1 = sourceRowsPerIteration == 2 && highBitDepthRowOwner is not null + ? highBitDepthRowOwner.GetSpan().Slice(image.Width, image.Width) + : Span.Empty; + for (int y = 0; y < image.Height; y += sourceRowsPerIteration) { - PixelOperations.Instance.ToRgb24( - configuration, - image.PixelBuffer.DangerousGetRowSpan(y), - rgbRow0); + if (isEightBit) + { + PixelOperations.Instance.ToRgb24( + configuration, + image.PixelBuffer.DangerousGetRowSpan(y), + rgbRow0); + } + else + { + // Rgb48 retains source component precision before the values are quantized to the requested + // 10-bit or 12-bit AV1 sample range. + PixelOperations.Instance.ToRgb48( + configuration, + image.PixelBuffer.DangerousGetRowSpan(y), + highBitDepthRgbRow0); + } bool hasSecondSourceRow = sourceRowsPerIteration == 2 && y + 1 < image.Height; if (hasSecondSourceRow) { - PixelOperations.Instance.ToRgb24( - configuration, - image.PixelBuffer.DangerousGetRowSpan(y + 1), - rgbRow1); + if (isEightBit) + { + PixelOperations.Instance.ToRgb24( + configuration, + image.PixelBuffer.DangerousGetRowSpan(y + 1), + rgbRow1); + } + else + { + PixelOperations.Instance.ToRgb48( + configuration, + image.PixelBuffer.DangerousGetRowSpan(y + 1), + highBitDepthRgbRow1); + } } - if (frameBuffer.BitDepth == Av1BitDepth.EightBit) + if (isEightBit) { Span yRow0 = yPlane.DangerousGetRowSpan(y); if (isMonochrome || subX == 0) @@ -242,7 +309,7 @@ internal static class Av1YuvConverter if (isMonochrome || subX == 0) { ConvertRgbToYuvRow( - rgbRow0, + highBitDepthRgbRow0, yRow0, isMonochrome ? Span.Empty : frameBuffer.GetHighBitDepthRowSpan(Av1Plane.U, y, 0, 0), isMonochrome ? Span.Empty : frameBuffer.GetHighBitDepthRowSpan(Av1Plane.V, y, 0, 0), @@ -259,8 +326,8 @@ internal static class Av1YuvConverter else { ConvertRgbToSubsampledYuvRows( - rgbRow0, - hasSecondSourceRow ? rgbRow1 : ReadOnlySpan.Empty, + highBitDepthRgbRow0, + hasSecondSourceRow ? highBitDepthRgbRow1 : ReadOnlySpan.Empty, yRow0, hasSecondSourceRow ? frameBuffer.GetHighBitDepthRowSpan(Av1Plane.Y, y + 1, 0, 0) : Span.Empty, frameBuffer.GetHighBitDepthRowSpan(Av1Plane.U, y >> subY, subX, subY), @@ -375,6 +442,7 @@ internal static class Av1YuvConverter /// Converts one YUV row to packed RGB using the resolved H.273 conversion state. /// /// The encoded sample type. + /// The packed RGB staging type. /// The luma samples. /// The upper blue-difference chroma row. /// The lower blue-difference chroma row. @@ -395,14 +463,14 @@ internal static class Av1YuvConverter /// The encoded chroma midpoint. /// The encoded chroma range. /// The largest encoded sample value. - private static void ConvertYuvToRgbRow( + private static void ConvertYuvToRgbRow( ReadOnlySpan ySource, ReadOnlySpan uRow0, ReadOnlySpan uRow1, ReadOnlySpan vRow0, ReadOnlySpan vRow1, int y1Weight, - Span destination, + Span destination, bool isMonochrome, int subX, int subY, @@ -417,6 +485,7 @@ internal static class Av1YuvConverter float chromaScale, float sampleMaximum) where TSample : unmanaged + where TRgb : unmanaged { for (int x = 0; x < destination.Length; x++) { @@ -460,10 +529,26 @@ internal static class Av1YuvConverter } } - destination[x] = new Rgb24( - ToSample(r * ByteMaximum, ByteMaximum), - ToSample(g * ByteMaximum, ByteMaximum), - ToSample(b * ByteMaximum, ByteMaximum)); + // The generic staging type is controlled by the frame bit depth. The JIT removes the inactive branch, + // retaining direct component access without routing every pixel through Vector4 or interface dispatch. + if (typeof(TRgb) == typeof(Rgb24)) + { + Rgb24 pixel = new( + ToSample(r * ByteMaximum, ByteMaximum), + ToSample(g * ByteMaximum, ByteMaximum), + ToSample(b * ByteMaximum, ByteMaximum)); + + destination[x] = Unsafe.As(ref pixel); + } + else + { + Rgb48 pixel = new( + ToSample(r * UShortMaximum, UShortMaximum), + ToSample(g * UShortMaximum, UShortMaximum), + ToSample(b * UShortMaximum, UShortMaximum)); + + destination[x] = Unsafe.As(ref pixel); + } } } @@ -545,6 +630,7 @@ internal static class Av1YuvConverter /// Converts one packed RGB row to luma and optional full-resolution chroma using the resolved H.273 conversion state. /// /// The encoded sample type. + /// The packed RGB staging type. /// The source RGB pixels. /// The destination luma samples. /// The destination blue-difference chroma samples. @@ -558,8 +644,8 @@ internal static class Av1YuvConverter /// The encoded chroma midpoint. /// The encoded chroma range. /// The largest encoded sample value. - private static void ConvertRgbToYuvRow( - ReadOnlySpan source, + private static void ConvertRgbToYuvRow( + ReadOnlySpan source, Span yDestination, Span uDestination, Span vDestination, @@ -573,6 +659,7 @@ internal static class Av1YuvConverter float chromaScale, float sampleMaximum) where TSample : unmanaged + where TRgb : unmanaged { for (int x = 0; x < source.Length; x++) { @@ -599,6 +686,7 @@ internal static class Av1YuvConverter /// Converts one or two packed RGB rows to luma and horizontally subsampled chroma. /// /// The encoded sample type. + /// The packed RGB staging type. /// The first source row. /// The optional second source row for 4:2:0 conversion. /// The first destination luma row. @@ -614,9 +702,9 @@ internal static class Av1YuvConverter /// The encoded chroma midpoint. /// The encoded chroma range. /// The largest encoded sample value. - private static void ConvertRgbToSubsampledYuvRows( - ReadOnlySpan sourceRow0, - ReadOnlySpan sourceRow1, + private static void ConvertRgbToSubsampledYuvRows( + ReadOnlySpan sourceRow0, + ReadOnlySpan sourceRow1, Span yDestination0, Span yDestination1, Span uDestination, @@ -631,6 +719,7 @@ internal static class Av1YuvConverter float chromaScale, float sampleMaximum) where TSample : unmanaged + where TRgb : unmanaged { int rowCount = sourceRow1.IsEmpty ? 1 : 2; for (int x = 0; x < sourceRow0.Length; x += 2) @@ -640,7 +729,7 @@ internal static class Av1YuvConverter float crSum = 0F; for (int row = 0; row < rowCount; row++) { - ReadOnlySpan source = row == 0 ? sourceRow0 : sourceRow1; + ReadOnlySpan source = row == 0 ? sourceRow0 : sourceRow1; Span yDestination = row == 0 ? yDestination0 : yDestination1; for (int column = 0; column < columnCount; column++) { @@ -666,6 +755,7 @@ internal static class Av1YuvConverter /// /// Converts one packed RGB pixel to normalized luma and chroma values. /// + /// The packed RGB staging type. /// The source RGB pixel. /// The conversion mode. /// The red luma coefficient. @@ -675,8 +765,8 @@ internal static class Av1YuvConverter /// The normalized blue-difference chroma result. /// The normalized red-difference chroma result. [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void ConvertRgbToYuv( - Rgb24 pixel, + private static void ConvertRgbToYuv( + TRgb pixel, ConversionMode mode, float kr, float kg, @@ -684,10 +774,29 @@ internal static class Av1YuvConverter out float y, out float cb, out float cr) + where TRgb : unmanaged { - float r = pixel.R / ByteMaximum; - float g = pixel.G / ByteMaximum; - float b = pixel.B / ByteMaximum; + float r; + float g; + float b; + + // These are the only staging formats selected by the owning conversion methods. Keeping the format choice + // generic lets the JIT specialize the hot loop and preserves high-bit-depth input without boxing or copies. + if (typeof(TRgb) == typeof(Rgb24)) + { + Rgb24 rgb24 = Unsafe.As(ref pixel); + r = rgb24.R / ByteMaximum; + g = rgb24.G / ByteMaximum; + b = rgb24.B / ByteMaximum; + } + else + { + Rgb48 rgb48 = Unsafe.As(ref pixel); + r = rgb48.R / UShortMaximum; + g = rgb48.G / UShortMaximum; + b = rgb48.B / UShortMaximum; + } + switch (mode) { case ConversionMode.Identity: