diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index 64f9509be..45d6a720a 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -67,7 +67,7 @@ This snapshot pins or classifies the available references and failures; it does | Managed implementation | Normative behavior | Reviewed implementation reference | Use | | --- | --- | --- | --- | -| `Av1YuvConverter.ConvertToRgb`, `ConvertFromRgb`, scalar row conversion, chroma reconstruction, and chroma downsampling | ITU-T H.273 (V4) equations 27-78, including chromaticity-derived equations 39-47 and constant-luminance equations 66-75; AV1 section 6.4.2 chroma sample positions | Official ITU-T H.273 (V4) (07/2024); libavif `src/reformat.c` and `src/colr.c` at `092276ce89098ead06db80975173191e5fee1826`; libaom `aom/aom_image.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar behavioral oracle for full/limited-range conversion at 8, 10, and 12 bits. Decode and encode cover identity, YCgCo, coefficient-based NCL, both fixed and chromaticity-derived constant/non-constant-luminance systems, and SMPTE ST 2085 across monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with AV1 chroma sample positioning. Chromaticity derivation uses every defined H.273 primary and matches libavif's BT.709 fallback for unspecified or reserved primaries. Later SIMD paths must preserve this scalar behavior. | +| `Av1YuvConverter.ConvertToRgb`, `ConvertFromRgb`, scalar row conversion, chroma reconstruction, and chroma downsampling | ITU-T H.273 (V4) equations 14-16 and 27-84, including chromaticity-derived equations 39-47, constant-luminance equations 66-75, and the PQ/HLG ICtCp matrices; AV1 section 6.4.2 chroma sample positions | Official ITU-T H.273 (V4) (07/2024); libavif `src/reformat.c` and `src/colr.c` at `092276ce89098ead06db80975173191e5fee1826`; libaom `aom/aom_image.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Scalar behavioral oracle for full/limited-range conversion at 8, 10, and 12 bits. Decode and encode cover identity, YCgCo, coefficient-based NCL, both fixed and chromaticity-derived constant/non-constant-luminance systems, SMPTE ST 2085, and ICtCp across monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with AV1 chroma sample positioning. Chromaticity derivation uses every defined H.273 primary and matches libavif's BT.709 fallback for unspecified or reserved primaries. The ICtCp inverse is derived from the exact H.273 integer matrices rather than an unrelated display conversion. Later SIMD paths must preserve this scalar behavior. | | `Av1TransferFunctions` | ITU-T H.273 (V4) Table 3 transfer characteristics 1-18 | Official ITU-T H.273 (V4) (07/2024); libavif `src/colr.c` at `092276ce89098ead06db80975173191e5fee1826` | Apply every AV1-signallable transfer function required by constant-luminance and ICtCp color conversion. Retain the H.273 normalized PQ and HLG definitions; do not import libavif's display-oriented 203-nit scaling or HLG OOTF into codec sample interpretation. Use libavif's midpoint convention only for the non-bijective zero code of the two logarithmic curves. | | `Av1FrameBuffer` high-bit-depth sample layout and `Av1YuvConverter` 10/12-bit packed-pixel conversion | AV1 section 6.4.1 bit depth and H.273 sample-range scaling | libaom `aom_scale/yv12config.h`, `av1/common/idct.c`, and `av1/common/reconintra.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f`; libavif `src/avif.c` and `src/reformat.c` at `092276ce89098ead06db80975173191e5fee1826` | Establish two-byte native sample storage with sample-unit strides for 10/12-bit reconstruction and use ImageSharp's existing `Rgb48` pixel-operation paths in both directions so packed-pixel staging does not reduce high-bit-depth samples to eight bits. | | `Av1PredictionDecoder`, `Av1HighBitDepthPredictor`, `Av1ChromaFromLumaContext`, `Av1PartitionInfo`, and the scalar DC, directional, Paeth, smooth, filter-intra, and chroma-from-luma predictors | AV1 sections 7.11.2 and 7.11.2.3 intra prediction | libaom `aom_dsp/intrapred.c`, `av1/common/reconintra.c`, `av1/common/av1_common_int.h`, `av1/common/blockd.h`, `av1/common/cfl.c`, and `av1/common/cfl.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Behavioral oracle for luma/chroma mode-neighbor addressing, directional upsampling, Paeth selection, smooth normalization, filter-intra taps, high-bit-depth clipping, chroma-from-luma storage/subsampling, and chroma-from-luma row strides. Existing managed scalar tables and predictors remain the implementation base. The WIP rectangular byte-pipeline smooth digest expectations encode width/height-swapped weights and must be replaced only from an independently generated oracle, not regenerated from this implementation. | @@ -131,7 +131,7 @@ This assessment is based on the current source after the upstream ImageSharp mer - Palette mode now reads the normative luma/chroma mode and size CDFs, neighbor color caches, high-bit-depth color syntax, diagonal color-index maps, clipped-edge padding, and direct sample prediction through the existing reconstruction pipeline. The scalar implementation matches the pinned libaom source, but an independently encoded palette AVIF fixture is still required before this path is independently verified. - Loop-restoration unit parsing records tile-local switchable/Wiener/self-guided filter selections and coefficients in frame-owned plane grids, including super-resolution-adjusted unit corners and the corrected conditional 64x64-superblock unit-size bit. The active restoration stage implements the normative unit geometry, striped deblocked boundaries, Wiener filtering, self-guided projection, and 8/10/12-bit clipping, while reusing the existing SIMD-backed super-resolution and adjacent multiply/add primitives. Independently encoded fixtures covering every parameter set, plane layout, bit depth, and frame-edge geometry are still required. Other normative independently decodable still-image syntax paths still contain `NotImplementedException` or equivalent unsupported branches. Tile-local palette CDF adaptation is present; the remaining still-image frame-context behavior requires a separate source audit without introducing sequence playback state. - The frame buffer now establishes two-byte native sample storage, logical plane rows, and sample-unit block strides for 10/12-bit frames. The active intra-prediction, inverse-transform, and block-reconstruction path selects native 16-bit samples for 10/12-bit frames and has focused pipeline wiring coverage. Chroma-from-luma storage, subsampling, parameter derivation, U/V sharing, and 8/10/12-bit prediction are active; independently encoded high-bit-depth and chroma-from-luma AVIF conformance files are still required. -- `Av1YuvConverter` now consumes the signaled range, H.273 matrix coefficients implemented so far, transfer characteristics where the matrix definition requires them, subsampling, and chroma sample position for 8, 10, and 12-bit output. Its high-bit-depth decode and encode paths use allocator-backed `Rgb48` rows and the existing `PixelOperations` conversions, avoiding the former eight-bit intermediate. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with libavif-compatible box averaging. Identity, YCgCo, the fixed non-constant-luminance matrices, both fixed and chromaticity-derived constant/non-constant-luminance systems, and SMPTE ST 2085 are active in both directions. ICtCp remains incomplete. Limited-range YCgCo also remains unsupported because the existing libavif-derived range pipeline scales YUV rather than the R/G/B signals required by H.273 equations 27-29. +- `Av1YuvConverter` now consumes the signaled range, every non-reserved AV1 H.273 matrix coefficient, transfer characteristics where the matrix definition requires them, subsampling, and chroma sample position for 8, 10, and 12-bit output. Its high-bit-depth decode and encode paths use allocator-backed `Rgb48` rows and the existing `PixelOperations` conversions, avoiding the former eight-bit intermediate. Encoder conversion covers monochrome, YUV 4:2:0, 4:2:2, and 4:4:4 with libavif-compatible box averaging. Identity, YCgCo, the fixed non-constant-luminance matrices, both fixed and chromaticity-derived constant/non-constant-luminance systems, SMPTE ST 2085, and PQ/HLG ICtCp are active in both directions. Limited-range YCgCo remains unsupported because the existing libavif-derived range pipeline scales YUV rather than the R/G/B signals required by H.273 equations 27-29. - The inverse-transform path allocates arrays in a per-transform hot path. - The production prediction, transform, and nonlinear self-guided paths remain predominantly scalar. Normative super-resolution and Wiener horizontal products now reuse ImageSharp's cross-platform adjacent multiply/add SIMD helper with exact scalar fallbacks; further SIMD work must preserve these scalar reconstruction oracles. diff --git a/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs b/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs index a41142f54..8d0f597f6 100644 --- a/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs +++ b/src/ImageSharp/Formats/Heif/Av1/Av1YuvConverter.cs @@ -54,6 +54,11 @@ internal static class Av1YuvConverter /// A constant-luminance transform using the signaled transfer characteristics. /// ConstantLuminance, + + /// + /// The BT.2100 ICtCp color transform. + /// + ICtCp, } /// @@ -454,6 +459,9 @@ internal static class Av1YuvConverter mode = ConversionMode.ConstantLuminance; GetChromaticityDerivedCoefficients(frameBuffer.ColorConfig.ColorPrimaries, out kr, out kb); break; + case ObuMatrixCoefficients.Bt2100ICtCp: + mode = ConversionMode.ICtCp; + break; default: throw new NotSupportedException($"AV1 matrix coefficients '{frameBuffer.ColorConfig.MatrixCoefficients}' are not currently supported."); } @@ -747,6 +755,52 @@ internal static class Av1YuvConverter g = Av1TransferFunctions.ToGamma(transferCharacteristics, linearGreen); b = nonlinearBlue; break; + case ConversionMode.ICtCp: + float nonlinearL; + float nonlinearM; + float nonlinearS; + if (transferCharacteristics == ObuTransferCharacteristics.Hlg) + { + // This is the exact inverse of H.273 equations 82 to 84. The first column is one + // because intensity is defined as the average of the L and M components. + nonlinearL = y + (0.015718580108730413F * cb) + (0.2095810681164055F * cr); + nonlinearM = y - (0.015718580108730413F * cb) - (0.2095810681164055F * cr); + nonlinearS = y + (1.0212710798422342F * cb) - (0.6052744909924315F * cr); + } + else + { + // H.273 equations 79 to 81 are the ICtCp matrix selected for PQ and every transfer + // code other than HLG. These constants are the exact inverse of its integer matrix. + nonlinearL = y + (0.008609037037932756F * cb) + (0.11102962500302596F * cr); + nonlinearM = y - (0.008609037037932756F * cb) - (0.11102962500302596F * cr); + nonlinearS = y + (0.5600313357106791F * cb) - (0.32062717498731885F * cr); + } + + float linearL = Av1TransferFunctions.ToLinear(transferCharacteristics, nonlinearL); + float linearM = Av1TransferFunctions.ToLinear(transferCharacteristics, nonlinearM); + float linearS = Av1TransferFunctions.ToLinear(transferCharacteristics, nonlinearS); + + // This cofactor inverse of H.273 equations 14 to 16 recovers linear RGB from LMS. + // Applying the transfer curve last returns the nonlinear RGB values stored by ImageSharp. + float ictcpLinearRed = + (3.4366066943330784F * linearL) - + (2.50645211865627F * linearM) + + (0.06984542432319148F * linearS); + + float ictcpLinearGreen = + (-0.7913295555989287F * linearL) + + (1.9836004517922907F * linearM) - + (0.192270896193362F * linearS); + + float ictcpLinearBlue = + (-0.025949899690592672F * linearL) - + (0.09891371471172644F * linearM) + + (1.1248636144023192F * linearS); + + r = Av1TransferFunctions.ToGamma(transferCharacteristics, ictcpLinearRed); + g = Av1TransferFunctions.ToGamma(transferCharacteristics, ictcpLinearGreen); + b = Av1TransferFunctions.ToGamma(transferCharacteristics, ictcpLinearBlue); + break; default: r = y + (2F * (1F - kr) * cr); g = y - (2F * ((kr * (1F - kr) * cr) + (kb * (1F - kb) * cb)) / kg); @@ -1092,6 +1146,38 @@ internal static class Av1YuvConverter cr = redDifference / (2F * (redDifference <= 0F ? constantLuminanceScales.NegativeRed : constantLuminanceScales.PositiveRed)); + break; + case ConversionMode.ICtCp: + float ictcpLinearRed = Av1TransferFunctions.ToLinear(transferCharacteristics, r); + float ictcpLinearGreen = Av1TransferFunctions.ToLinear(transferCharacteristics, g); + float ictcpLinearBlue = Av1TransferFunctions.ToLinear(transferCharacteristics, b); + + // H.273 equations 14 to 16 convert linear BT.2100 RGB into the LMS cone-response domain + // before the signaled transfer curve is applied to each component. + float nonlinearL = Av1TransferFunctions.ToGamma( + transferCharacteristics, + ((1688F * ictcpLinearRed) + (2146F * ictcpLinearGreen) + (262F * ictcpLinearBlue)) / 4096F); + + float nonlinearM = Av1TransferFunctions.ToGamma( + transferCharacteristics, + ((683F * ictcpLinearRed) + (2951F * ictcpLinearGreen) + (462F * ictcpLinearBlue)) / 4096F); + + float nonlinearS = Av1TransferFunctions.ToGamma( + transferCharacteristics, + ((99F * ictcpLinearRed) + (309F * ictcpLinearGreen) + (3688F * ictcpLinearBlue)) / 4096F); + + y = 0.5F * (nonlinearL + nonlinearM); + if (transferCharacteristics == ObuTransferCharacteristics.Hlg) + { + cb = ((3625F * nonlinearL) - (7465F * nonlinearM) + (3840F * nonlinearS)) / 4096F; + cr = ((9500F * nonlinearL) - (9212F * nonlinearM) - (288F * nonlinearS)) / 4096F; + } + else + { + cb = ((6610F * nonlinearL) - (13613F * nonlinearM) + (7003F * nonlinearS)) / 4096F; + cr = ((17933F * nonlinearL) - (17390F * nonlinearM) - (543F * nonlinearS)) / 4096F; + } + break; default: y = (kr * r) + (kg * g) + (kb * b);