From a07bad340875fb9b19db0db9e5c0fd73256f317b Mon Sep 17 00:00:00 2001 From: James Jackson-South Date: Tue, 25 Aug 2026 05:27:05 +1000 Subject: [PATCH] Constrain HEVC work to still image items --- HEIF_IMPLEMENTATION_PLAN.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/HEIF_IMPLEMENTATION_PLAN.md b/HEIF_IMPLEMENTATION_PLAN.md index f62035a6a..b711f456b 100644 --- a/HEIF_IMPLEMENTATION_PLAN.md +++ b/HEIF_IMPLEMENTATION_PLAN.md @@ -59,12 +59,12 @@ Use the references in this order when behavior differs: 1. The published ISO BMFF, HEIF, HEVC, AV1, AV1-ISOBMFF, and AVIF requirements are normative. 2. The official [AOM AV1 Codec Library](https://aomedia.googlesource.com/aom/) is the primary implementation reference for AV1 decode, encode, high-bit-depth behavior, tests, and optimized scalar/SIMD algorithms. Pin one reviewed commit before porting. Use its scalar C paths as behavioral references and its architecture-specific paths as SIMD references to be expressed with ImageSharp's existing managed intrinsics. 3. The local `D:\GitHub\AOMediaCodec\libavif` checkout is the AVIF container, metadata, color-conversion, grid, alpha, and interoperability oracle. At inspection time it identifies itself as 1.4.2-devel. -4. A separately reviewed, license-compatible HEVC implementation reference must be pinned before HEVC porting begins. The HEVC specification remains normative, and independent HEIC software is required as an interoperability oracle. Do not copy from GPL or otherwise incompatible sources. +4. The official HM reference software at commit `9c1f298659ab0cee9dc13d23d0304221575410b9` is the primary scalar implementation reference for HEVC syntax, reconstruction, and the complete Range Extensions profile matrix. Android `libhevc` v1.6.0 at commit `c83a76b084498d55f252f48b2e3786804cdf24b7` is a secondary production-C comparison for its supported common 8-bit 4:2:0 paths; its decoder rejects other chroma formats and bit depths and therefore is not the feature-coverage authority. The HEVC specification remains normative, and an independent HEIC container implementation is still required as an interoperability oracle. Do not copy from GPL or otherwise incompatible sources. 5. Existing ImageSharp codecs are the authority for ImageSharp API shape, memory ownership, stream behavior, cancellation, resource limits, pixel conversion, tests, and SIMD dispatch. The linked ImageSharp discussion establishes the project constraint: the shipped implementation is purely managed and other codec libraries are references, not native runtime dependencies. libaom is the official encoder/decoder implementation reference for AV1, but it does not parse the HEIF container or implement HEVC. libavif dispatches AV1 work to external codec libraries, so it remains an observable AVIF/container oracle rather than the source for every AV1 algorithm. -The AOM source is distributed under the BSD 2-Clause License and the Alliance for Open Media Patent License 1.0. Before porting further code, record the exact upstream file, commit, applicable license/patent notice, and corresponding managed file or method. Audit the existing SVT-AV1-attributed WIP separately rather than relabeling it as libaom-derived. Update `THIRD-PARTY-NOTICES.TXT` before any referenced implementation code is merged. A pure managed HEVC implementation does not remove HEVC patent or licensing obligations, so those must be resolved before the HEIC work is considered releasable. +The AOM source is distributed under the BSD 2-Clause License and the Alliance for Open Media Patent License 1.0. HM is distributed under its BSD license with an explicit patent-rights disclaimer, while the reviewed Android `libhevc` source is distributed under Apache-2.0. Before porting further code, record the exact upstream file, commit, applicable license/patent notice, and corresponding managed file or method. Audit the existing SVT-AV1-attributed WIP separately rather than relabeling it as libaom-derived. Update `THIRD-PARTY-NOTICES.TXT` before any referenced implementation code is merged. A pure managed HEVC implementation does not remove HEVC patent or licensing obligations, so those must be resolved before the HEIC work is considered releasable. ### Pinned reference and baseline snapshot @@ -72,12 +72,14 @@ The initial post-merge snapshot was established on 2026-08-24: - the official libaom reference is tag `v3.14.1`, commit `03087864cf4bea6abb0d28f95cf7843511413d8f`, matching the revision selected by the local libavif `ext/aom.cmd` dependency script; - the local libavif container, color-conversion, and interoperability oracle is commit `092276ce89098ead06db80975173191e5fee1826`, described as `v1.4.2-66-g092276ce`; +- the official HM reference software is commit `9c1f298659ab0cee9dc13d23d0304221575410b9`; its decoder and profile tables cover the HEVC Range Extensions chroma, bit-depth, intra, and still-picture profiles required for the final HEIC feature matrix; +- the official Android `libhevc` reference is v1.6.0 at commit `c83a76b084498d55f252f48b2e3786804cdf24b7`; it remains a secondary common-path comparison because its SPS parser accepts only 8-bit YUV 4:2:0; - the official ITU-T H.274 (V4) recommendation from January 2026 is the normative semantics reference for mastering-display and content color-volume fields reused by still-image item properties; its video-SEI persistence and cancellation behavior is outside this container scope; -- the independently reviewed HEVC implementation and interoperability references remain unresolved and must be pinned before HEVC algorithm work begins; +- an independent HEIC container interoperability oracle and the complete external HEIC fixture-generation matrix remain to be pinned before HEIC support can pass its final integration gate; - `dotnet build ImageSharp.sln -c Release --no-restore -m:1 -v minimal` succeeds with no errors after the upstream compatibility fixes; and - the existing HEIF-focused test run executes 8,198 cases, with 8,184 passing and 14 failing. Thirteen failures are isolated to the WIP AV1 YUV conversion tests, and one is the existing legacy JPEG HIF reference-image mismatch. Golden artifacts have not been changed. -This snapshot pins or classifies the available references and failures; it does not complete Phase 0. The full WIP provenance map, disabled-test inventory, HEVC reference selection, and feature-state matrix remain required. +This snapshot pins or classifies the available references and failures; it does not complete Phase 0. The full WIP provenance map, disabled-test inventory, HEIC container interoperability selection, and feature-state matrix remain required. ### Provenance map in progress @@ -109,7 +111,7 @@ This snapshot pins or classifies the available references and failures; it does | `HeifDecoderCore` color-property parsing/association, `HeifItem` color profiles, and `Av1Decoder` effective color validation | ISO/IEC 14496-12 section 12.1.5 color information; HEIF section 6.5.5.1 color-information properties; AV1-ISOBMFF section 2.3.4 configuration semantics | libavif `src/read.c` functions `avifParseColourInformationBox`, `avifReadColorNclxProperty`, and `avifReadColorProperties`, plus `src/write.c` function `avifEncoderWriteColorProperties`, at `092276ce89098ead06db80975173191e5fee1826` | Associate at most one ICC and one `nclx` property with each presented color image item, validate ICC payloads and CICP reserved bits, expose them through ImageSharp's existing profile types, and inherit a grid's CICP description only for tiles that do not declare one. For AV1 items, require every specified sequence-header CICP field and the range flag to match `nclx`; use the item property only to supply sequence-header fields coded as unspecified before color conversion, then expose the effective CICP values even when no item property is present. Retain only the two image color profiles; do not add a reusable color-box, sample-entry, track, or media model. | | `HeifPixelAspectRatio`, `HeifItem.PixelAspectRatio`, and `HeifDecoderCore.ApplyItemPixelAspectRatioMetadata` | ISO/IEC 14496-12 section 12.1.4.3 pixel aspect ratio | libavif `src/read.c` function `avifParsePixelAspectRatioBox`, `src/write.c` function `avifEncoderWritePaspProperty`, and the presented-image property selection in `src/read.c` at `092276ce89098ead06db80975173191e5fee1826` | Preserve the two unsigned 32-bit relative spacings on the associated image item, reject zero or duplicate ratios, and map the displayed pixel width-to-height ratio into ImageSharp's existing unitless resolution metadata. Exchange the metadata axes after a quarter-turn presentation rotation and fall back from a derived grid to its first decodable tile only when the grid does not declare `pasp`. This remains one still-image presentation property and introduces no generic transform, sample-entry, or display model. | | `Av1CodecConfiguration`, `HeifItem.Av1CodecConfiguration`, `Av1HeifItemDecoder`, and AV1 grid configuration checks | AV1-ISOBMFF sections 2.3.3 and 2.3.4 codec-configuration record syntax and semantics; AVIF sections 2.1, 2.2.1, and 2.2.3 AV1 image-item, item-configuration, and HDR metadata constraints; AV1 sections 5.8.3, 5.8.4, 6.7.3, and 6.7.4 HDR metadata syntax and semantics; ISOBMFF mastering-display and content-light image properties; ITU-T H.274 section 8.9 mastering-display field semantics; MIAF section 7.3.11.4.1 grid input constraints | libavif `src/read.c` functions `avifParseCodecConfiguration`, `avifDecoderItemValidateProperties`, `avifReadCodecConfigProperty`, `avifParseContentLightLevelInformation`, and `avifSkipMasteringDisplayColourVolume` at `092276ce89098ead06db80975173191e5fee1826`; libaom `av1/decoder/obu.c` functions `read_metadata`, `read_metadata_hdr_cll`, and `read_metadata_hdr_mdcv` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Associate exactly one `av1C` property with each decoded `av01` image item, validate the fixed record and its bit depth/chroma fields against the item's AV1 sequence header and optional `pixi` channel depths, require matching configurations across grid tiles, and report the encoded image precision and monochrome shape through `HeifMetadata`. Validate low-overhead OBU framing, require exactly one sequence header in the image item, allow at most one first-position sequence header in `configOBUs`, and compare a repeated header's extension and payload exactly while ignoring only its legal size-field representation. Decode `clli` and `mdcv` as still-image item properties, validate matching HDR CLL and HDR MDCV metadata OBUs from the combined configuration/item sequence, and account for the different primary order and fixed-point precision of the ISOBMFF and AV1 MDCV representations. Expose the effective HDR values without adding sample groups, tracks, or media metadata. Related still-image HDR properties remain required. Consume but do not retain presentation-delay syntax, and introduce no sample entry, sample description, track, timing, or generic decoder-configuration model. | -| `HevcCodecConfiguration`, `HevcNalUnitArray`, and `HeifItem.HevcCodecConfiguration` | ISO/IEC 14496-15 `HEVCDecoderConfigurationRecord`, HEVC NAL-unit header syntax, and HEVC image-item configuration requirements | Android `libhevc` v1.6.0 decoder NAL and parameter-header paths at `c83a76b084498d55f252f48b2e3786804cdf24b7`; independently produced HEIC fixtures for `hvcC` record layout | Associate exactly one `hvcC` property with each `hvc1` image item; validate the fixed profile, level, chroma, bit-depth, reserved, length-size, array, and NAL-header fields within the property boundary; and retain only the image description plus bounded configuration NAL units required to decode that item. Consume but do not retain average-frame-rate or temporal-layer presentation fields. Do not add visual sample entries, sample descriptions, tracks, timing, access-unit timelines, or a generic decoder-configuration model. Apache-2.0 notice preservation and separate HEVC patent/release clearance remain final integration gates. | +| `HevcCodecConfiguration`, `HevcNalUnitArray`, and `HeifItem.HevcCodecConfiguration` | ISO/IEC 14496-15 `HEVCDecoderConfigurationRecord`, HEVC NAL-unit header syntax, and HEVC image-item configuration requirements | HM `source/Lib/TLibDecoder/NALread.cpp` and parameter-header paths at `9c1f298659ab0cee9dc13d23d0304221575410b9`; Android `libhevc` v1.6.0 decoder NAL and parameter-header paths at `c83a76b084498d55f252f48b2e3786804cdf24b7`; independently produced HEIC fixtures for `hvcC` record layout | Associate exactly one `hvcC` property with each `hvc1` image item; validate the fixed profile, level, chroma, bit-depth, reserved, length-size, array, and NAL-header fields within the property boundary; and retain only the image description plus bounded configuration NAL units required to decode that item. Consume but do not retain average-frame-rate or temporal-layer presentation fields. Do not add visual sample entries, sample descriptions, tracks, timing, access-unit timelines, or a generic decoder-configuration model. BSD and Apache-2.0 notice preservation and separate HEVC patent/release clearance remain final integration gates. | | `HeifContentColorVolume`, `HeifItem.ContentColorVolume`, and `HeifDecoderCore` content color-volume parsing and presentation | HEIF content color-volume item property; AVIF 1.2 content color-volume requirements; ITU-T H.274 (V4) content colour volume syntax and semantics | libavif `src/read.c` function `avifSkipContentColourVolume` at `092276ce89098ead06db80975173191e5fee1826`; official ITU-T H.274 (V4), January 2026 | Decode only the bounded per-image `cclv` property: require zero cancellation, persistence, and reserved bits; preserve optional signed G/B/R primary coordinates and normalized minimum, maximum, and average luminance values; and validate their registered ranges and ordering. Expose the effective grid-or-tile still-image value through `HeifMetadata`. Do not add SEI persistence, retained video state, tracks, samples, timing, or a generic ISO BMFF color-volume box model. | | `HeifAmbientViewingEnvironment`, `HeifReferenceViewingEnvironment`, `HeifNominalDiffuseWhite`, and their per-item presentation metadata | ISOBMFF ambient viewing environment; ITU-T H.274 (V4) section 8.13; HEIF Amendment 1 sections 6.5.44 and 6.5.45; AVIF 1.2 image-item box requirements | libavif `src/read.c` functions `avifSkipAmbientViewingEnvironment`, `avifSkipReferenceViewingEnvironment`, and `avifSkipNominalDiffuseWhite` at `092276ce89098ead06db80975173191e5fee1826`; official ITU-T H.274 (V4), January 2026 | Decode the fixed-size `amve`, version-zero `reve`, and version-zero `ndwt` properties only when associated with the presented still-image item. Preserve physical illuminance and luminance units, distinct surround/periphery chromaticities, and the coded zero that requests the standard nominal diffuse-white default. Validate registered coordinate ranges and duplicate associations. Retain no video-SEI persistence, visual sample entry, display pipeline, track, timing, or generic viewing-environment box model. | | `GridHeifItemDecoder` and `HeifDecoderCore` grid/thumbnail selection | ISO/IEC 23008-12 section 6.6.2.3 image-grid syntax and MIAF grid-cell constraints | libavif `src/read.c` functions `avifParseImageGridBox`, `avifDecoderDataAllocateImagePlanes`, and `avifDecoderDataCopyTileToImage` at `092276ce89098ead06db80975173191e5fee1826` | Parse version-zero 16-bit and 32-bit grid descriptors, preserve row-major `dimg` order, require the declared tile count and one coding format, validate canvas coverage and edge overlap, and crop only the rightmost column and bottom row while copying through ImageSharp pixel buffers. A primary grid whose tile codec is unavailable may use only a decodable thumbnail that explicitly references that grid. | @@ -143,14 +145,14 @@ This assessment is based on the current source after the upstream ImageSharp mer - Decode now resolves `cdsc`-associated Exif and `application/rdf+xml` XMP items for the primary still image, validates the declared Exif TIFF-header offset, and attaches the payloads through ImageSharp's existing profile types before presentation transforms. Independent AVIF, HEIC, and HIF metadata fixtures and Identify-time profile reporting remain required. - ICC and `nclx` CICP color properties are now associated with the presented color item instead of global parser state, validated, and exposed on Decode and Identify through the existing ImageSharp profiles. AV1 decode requires specified sequence-header fields and the range flag to match `nclx`, uses the property only for fields the bitstream leaves unspecified, and exposes the effective sequence-header CICP description when no property is present. Independent ICC/CICP fixtures, Identify-time bitstream fallback, ICC conversion coverage, and HEVC integration remain required. - Pixel aspect ratio now preserves the complete unsigned spacing pair, affects Decode and Identify through ImageSharp's existing unitless resolution metadata, and follows quarter-turn presentation rotation. Independent grid, rotation, and maximum-spacing fixtures remain required. -- AV1 codec configuration is now retained per `av01` image item rather than in decoder-global state. Decode requires the property, validates its fixed record against the item's sequence header and any associated `pixi` channel depths, and requires matching configurations across grid tiles. Identify now reports the configuration's 8/10/12-bit precision and monochrome shape. The optional `configOBUs` sequence is bounded and validated, including its mandatory size fields, first-position/at-most-one sequence-header rule, the image item's exactly-one sequence-header rule, and exact comparison of a repeated configuration header with the item header. Content light-level and mastering-display color-volume information are decoded from the bounded `clli` and `mdcv` image properties and matching AV1 HDR metadata OBUs, with the representations' distinct fixed-point precision, grid/property precedence, and `SkipMetadata` behavior preserved. The still-image `cclv`, `amve`, `reve`, and `ndwt` properties preserve content-volume, ambient-viewing, mastering-environment, and diffuse-white descriptions without importing their video-SEI or sample-entry state models. Independent fixtures and encoder preservation remain required; HEVC `hvcC` remains unimplemented. +- AV1 codec configuration is now retained per `av01` image item rather than in decoder-global state. Decode requires the property, validates its fixed record against the item's sequence header and any associated `pixi` channel depths, and requires matching configurations across grid tiles. Identify now reports the configuration's 8/10/12-bit precision and monochrome shape. The optional `configOBUs` sequence is bounded and validated, including its mandatory size fields, first-position/at-most-one sequence-header rule, the image item's exactly-one sequence-header rule, and exact comparison of a repeated configuration header with the item header. Content light-level and mastering-display color-volume information are decoded from the bounded `clli` and `mdcv` image properties and matching AV1 HDR metadata OBUs, with the representations' distinct fixed-point precision, grid/property precedence, and `SkipMetadata` behavior preserved. The still-image `cclv`, `amve`, `reve`, and `ndwt` properties preserve content-volume, ambient-viewing, mastering-environment, and diffuse-white descriptions without importing their video-SEI or sample-entry state models. Independent fixtures and encoder preservation remain required. HEVC `hvcC` is now associated with each `hvc1` image item, structurally validated, and used to report encoded precision and monochrome shape; parameter-set validation and pixel reconstruction remain incomplete. - Several image-item properties and relationships remain missing or parsed without fully affecting output. - Identify and decode now use the same bounded metadata parser and both validate the complete leading file type box. The parsed state is still mutable and Identify does not yet report the complete bit depth, color, profile, or transform model. ### HEVC decoder and encoder -- `Heif4CharCode` recognizes `hvc1` image items and `hvcC` configuration, and Identify classifies HEVC fixtures, but `HeifCompressionFactory` has no HEVC item decoder. -- There is no HEVC bitstream parser, CABAC decoder, coding-tree reconstruction, still-image intra prediction, inverse transform, deblocking, sample-adaptive offset, or high-bit-depth path. +- `Heif4CharCode` recognizes `hvc1` image items, `HevcCodecConfiguration` validates and associates the bounded `hvcC` property, and Identify reports its HEVC precision and monochrome shape, but `HeifCompressionFactory` has no HEVC item decoder. +- There is no HEVC NAL/RBSP or parameter-set parser, CABAC decoder, coding-tree reconstruction, still-image intra prediction, inverse transform, deblocking, sample-adaptive offset, or high-bit-depth reconstruction path. - There is no HEVC encoder. The current HEIC-branded encoder writes a legacy JPEG payload and therefore cannot provide HEIC output. - Existing HEVC tests prove container identification only; they do not decode or compare HEIC pixels. @@ -251,7 +253,7 @@ Tasks: 1. Build the merged solution in Release and record compile errors and warnings attributable to the WIP. 2. Run only the existing HEIF/HEVC/AV1 tests first, then record disabled tests and unexecuted asset coverage. -3. Pin an official libaom commit for AV1, the local libavif commit for AVIF/container comparison, and the independently reviewed HEVC implementation and interoperability references. +3. Record the pinned official libaom, local libavif, HM, and Android `libhevc` revisions, then pin the independent HEIC container interoperability oracle and external fixture toolchain. 4. Create a provenance map from each WIP codec file to its specification section and exact upstream source. Preserve the current SVT-AV1 origins where applicable and identify which missing paths will use libaom. 5. Convert the completion boundary above into a feature matrix with `unsupported`, `parses`, `decodes`, `encodes`, and `verified independently` states. 6. Audit every source file under `src/ImageSharp/Formats/Heif`. Document every type and shared contract, and add technical comments wherever the code depends on non-obvious specification syntax, fixed-point arithmetic, transform staging, entropy state, buffer layout, or SIMD behavior. Keep public XML documentation limited to observable API behavior. @@ -318,8 +320,8 @@ Exit gate: Implement and verify in dependency order: -1. HEVC byte-stream and length-delimited NAL units, `hvcC`, VPS, SPS, PPS, access units, and the parameter-set and slice-header syntax permitted for independently decodable still-image items. -2. One coherent decoder lifecycle that owns parameter sets, picture state, slice/tile entropy state, and reconstructed planes. +1. HEIF item-local length-delimited NAL units, `hvcC`, VPS, SPS, PPS, and the parameter-set and slice-header syntax needed to decode the one independently coded picture carried by a supported still-image item. Do not add an Annex B byte-stream API or an access-unit/timeline abstraction. +2. One bounded decoder lifecycle that owns only the parameter sets, current-picture state, slice/tile entropy state, and reconstructed planes required by that image item. 3. CABAC arithmetic decoding and every required context transition. 4. Coding-tree, coding-unit, prediction-unit, and transform-unit traversal across all permitted sizes and partition modes. 5. Intra prediction for every luma and chroma mode, including strong intra smoothing and constrained prediction rules.