Browse Source

Record JPEG-style HEIF operator architecture

pull/2633/head
James Jackson-South 4 days ago
parent
commit
438fd6f2bf
  1. 10
      HEIF_IMPLEMENTATION_PLAN.md

10
HEIF_IMPLEMENTATION_PLAN.md

@ -29,9 +29,9 @@ Checkboxes may be marked complete only when the implementation and the verificat
## Delivery dashboard
Last reconciled with the source tree on 2026-08-30 against production checkpoint `86d44f158af66e6d105a39f1fc2854c56270d8bb`. Committed checkpoints include the AV1 transform architecture, OBU framing, tile partitioning, mode information, segmentation, delta quantization, transform-size selection, coefficient decoding, inverse quantization, inverse transforms, complete intra prediction, intra-block copy, official extreme-displacement intra-block-copy conformance, the 12-profile reconstruction matrix, corrected twelve-bit inverse-transform SIMD arithmetic, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, official spatial- and temporal-layer default-operating-point conformance, official active-film-grain and monochrome sequence conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
Last reconciled with the source tree on 2026-08-30 against production checkpoint `a0d1b5af56acb3a3a9aebb29c916d1b36cd49367`. Committed checkpoints include the AV1 transform architecture, OBU framing, tile partitioning, mode information, segmentation, delta quantization, transform-size selection, coefficient decoding, inverse quantization, inverse transforms, complete intra prediction, intra-block copy, official extreme-displacement intra-block-copy conformance, the 12-profile reconstruction matrix, corrected twelve-bit inverse-transform SIMD arithmetic, layered-item properties, layered reference/header/CDF/motion-field state, inter-frame intra blocks, SIMD-first translational prediction, complete single-reference inter reconstruction, compound reference trees and modes, paired reference-MV derivation, reference-dependent bounded sequence decoding, allocation-free SIMD-first equal averaging, selected inter-intra prediction, selectable compound blending, OBMC, scaled-reference reconstruction, local warped prediction, non-translational global prediction, official motion-vector conformance, official spatial- and temporal-layer default-operating-point conformance, official active-film-grain and monochrome sequence conformance, selected spatial-layer presentation, and progressive color and auxiliary-alpha conformance. This dashboard is the authoritative delivery order. The detailed phase checklists below provide subsystem evidence; they do not override the current-stage marker or permit work to skip ahead.
Commit `86d44f158af66e6d105a39f1fc2854c56270d8bb` restores the established AV1 transform operator architecture after the incorrect wrapper-owned layout. All twelve forward one-dimensional transforms are top-level semantic operators under `Transform.Forward`; all twelve inverse one-dimensional transforms are top-level semantic operators under `Transform.Inverse`; their static-generic contracts, the inverse reconstruction-output contract, and `Av1InverseTransformOutputOperator<TSample>` are top-level types. `Av1ForwardTransformer` and `Av1Inverse2dTransformer` now contain only the separate two-dimensional traversal and dispatch responsibilities, and `Av1InverseTransformer` remains the reconstruction facade. Normalized comparison proves that all 24 one-dimensional operator bodies and the output-operator body retain the preceding checkpoint's arithmetic exactly. Roslynk reports zero compiler errors, all 647 focused `net10.0` transform and transform-size cases pass without failures or skips, both Release source targets build with zero warnings and errors, every new C# path resolves to the existing `csharp` diff driver without an attributes change, and `git diff --check` is clean.
Commit `a0d1b5af56acb3a3a9aebb29c916d1b36cd49367` restores the established JPEG color-converter operator architecture throughout the HEIF implementation. `Av1ForwardTransformer` owns its nested one-dimensional contract and all twelve concrete forward transform operators; `Av1Inverse2dTransformer` owns the corresponding inverse contract and operators; and `Av1InverseTransformer` owns the byte and high-bit-depth reconstruction-output operators. Each contract lives in the owner's `.Operator.cs`, while every semantic transform with a shared traversal has its own `<Owner>.<Semantic>Operator.cs` partial-family file and implements concrete scalar and SIMD overload bodies without a forwarding semantic core. The same owner-and-partial-file rule now covers HEVC inverse transforms and intra prediction, AV1 and HEVC deblocking, AV1 CDEF, and HEVC transform-skip reconstruction. Vector-width and sample-storage adapters that are not semantic dispatch operators are named `Operations`; no operator type is named after a hardware width, no static contract uses CRTP, no source file groups multiple concrete semantic operator structs, and no semantic operator remains top-level. Roslynk reports zero compiler errors, both Release source targets build with zero warnings and errors, and 736 focused `net10.0` transform, loop-filter, prediction, reconstruction, transfer-function, and film-grain cases pass without failures or skips through the existing `FeatureTestRunner` coverage. All 56 new C# paths resolve to the existing `csharp` diff driver without an attributes change, and `git diff --check` is clean.
Commit `1c58d855f70b024170ced9eb0a7005f0f9c955ad` records the complete official motion-vector conformance checkpoint. The official IVF has SHA-1 `F064290D7FCD3B3DE19020E8AEC6C43C88D3A505`, matching the pinned libaom test-data manifest, and SHA-256 `222A9050059B254DAB17CFB802FF829C778E3F93AF18961A622C8268576C1395`; its pinned-libaom Y4M has SHA-256 `D97AC78C81782CF1507549368047769DC677DBE205706458D1EE9C807DE6EC78`. Both source targets build with zero warnings and errors, the `net10.0` test-project analyzer build completes with zero errors and 1,014 pre-existing repository warnings, Roslynk reports zero compiler errors, 3,985 focused `net10.0` cases pass without failures or skips, and `git diff --check` is clean.
@ -355,10 +355,10 @@ This snapshot pins or classifies the available references and failures; it does
| `Av1WienerFilter` | AV1 sections 7.17.4 and 7.17.5 Wiener restoration filtering and coefficient derivation | libaom `av1/common/restoration.c`, `av1/common/restoration.h`, `av1/common/convolve.c`, and `av1/common/convolve.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the implicit center-sample contribution, separable horizontal/vertical rounding, bit-depth-dependent 16-bit intermediate range, and final 8/10/12-bit clipping. Reuse `Vector128_.MultiplyAddAdjacent` for the contiguous horizontal eight-tap product with an exact scalar fallback. Keep the restoration stage disabled until stripe boundaries and self-guided filtering are both complete. |
| `Av1SelfGuidedFilter` | AV1 sections 7.17.2 and 7.17.3 self-guided and box-filter processes | libaom `av1/common/restoration.c` and `av1/common/restoration.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Design the window layout, caller-owned scratch, local statistics, and projection traversal from libaom's architecture-specific kernels so the production path is SIMD-first. Implement the equivalent scalar fallback through the same contract. Preserve the sixteen normative radius/variance parameter sets, local mean and variance normalization, alternating-row radius-two optimization, decoded projection-coefficient behavior, signed rounding, and 8/10/12-bit clipping. Keep this image-reconstruction stage disabled until restoration stripe boundaries are complete. |
| `Av1LoopRestorationBoundary`, `Av1LoopRestorationDecoder`, and `Av1FrameDecoder` restoration-stage ordering | AV1 section 7.17 loop restoration, including striped boundary semantics | libaom `av1/common/restoration.c`, `av1/common/restoration.h`, `av1/common/resize.c`, and `av1/decoder/decodeframe.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve two deblocked rows at internal 64-luma stripe boundaries before CDEF, apply the existing normative SIMD-backed super-resolution kernel to saved rows when scaled, use post-CDEF/super-resolution samples at frame edges, extend the final restoration unit up to 150 percent of nominal size, and filter from immutable plane snapshots into separate output planes. This is bounded still-image reconstruction state, not retained reference-frame, track, timing, or playback state. |
| `Av1FilmGrainDecoder`, `Av1FilmGrainGaussianSequence`, `Av1FilmGrainNoise`, `Av1FilmGrainOverlap`, and `Av1FilmGrainSampleOperator<TSample>` | AV1 section 7.18 film-grain synthesis | libaom `av1/decoder/grain_synthesis.c`, `av1/decoder/grain_synthesis.h`, and `aom_dsp/grain_params.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the normative 2,048-sample Gaussian sequence as compile-time span data, the serial linear-feedback shift register and luma/chroma autoregressive templates, scaling lookup interpolation, 32x32 block selection, boundary overlap, restricted-range clipping, monochrome and 4:2:0/4:2:2/4:4:4 layouts, and 8/10/12-bit arithmetic. Use allocator-owned scratch. Apply scaling and noise through AVX2 gather arithmetic at every bit depth, retain the measured cross-platform 128-bit high-bit-depth path, and use the exact scalar path for 8-bit machines without AVX2 and for vector tails. Process horizontal overlap through preferred-native 512-bit, then 256-bit, 128-bit, and scalar tiers; retain scalar vertical overlap because each output row exposes only one or two strided samples. Keep SIMD width, ISA, sample storage, and bit depth out of folders, namespaces, files, and type names. Apply grain only to displayed samples after all in-loop filters, inherit reference-selected parameters through the bounded image-layer decoder state, and keep the ungrained reconstruction in every refreshed reference slot. This does not add sequence playback or video scope. |
| `Av1FilmGrainDecoder`, `Av1FilmGrainGaussianSequence`, `Av1FilmGrainNoise`, `Av1FilmGrainOverlap`, and `Av1FilmGrainSampleOperations<TSample>` | AV1 section 7.18 film-grain synthesis | libaom `av1/decoder/grain_synthesis.c`, `av1/decoder/grain_synthesis.h`, and `aom_dsp/grain_params.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the normative 2,048-sample Gaussian sequence as compile-time span data, the serial linear-feedback shift register and luma/chroma autoregressive templates, scaling lookup interpolation, 32x32 block selection, boundary overlap, restricted-range clipping, monochrome and 4:2:0/4:2:2/4:4:4 layouts, and 8/10/12-bit arithmetic. Use allocator-owned scratch. Apply scaling and noise through AVX2 gather arithmetic at every bit depth, retain the measured cross-platform 128-bit high-bit-depth path, and use the exact scalar path for 8-bit machines without AVX2 and for vector tails. Process horizontal overlap through preferred-native 512-bit, then 256-bit, 128-bit, and scalar tiers; retain scalar vertical overlap because each output row exposes only one or two strided samples. Keep SIMD width, ISA, sample storage, and bit depth out of folders, namespaces, files, and type names. Apply grain only to displayed samples after all in-loop filters, inherit reference-selected parameters through the bounded image-layer decoder state, and keep the ungrained reconstruction in every refreshed reference slot. This does not add sequence playback or video scope. |
| `Av1FrameInfo`, `Av1TileReader`, and `Av1BlockDecoder` transform/coefficient storage | AV1 section 5.11.39 coefficient syntax and section 7.11.2 reconstruction | libaom `av1/decoder/decodetxb.c` and `av1/decoder/decoder.h` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve separate luma and chroma transform coefficients at monotonically advancing per-plane offsets within each superblock so reconstruction consumes the same transform-block order produced by tile parsing. |
| `Av1InverseQuantizer` and `Av1InverseQuantizationLookup` | AV1 section 7.12.3 inverse quantization | libaom `aom_dsp/aom_dsp_common.h`, `av1/common/quant_common.c`, and `av1/decoder/decodetxb.c` at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Select the per-segment matrix level, alias 64-pixel transform dimensions to their adjusted matrices, retain a flat level-15 matrix, and apply the five-bit inverse-matrix weight scale. The large managed lookup remains a single process-wide table. |
| `Av1ForwardTransformer`, `Av1Inverse2dTransformer`, `Av1Transform2dFlipConfiguration`, the forward/inverse 1-D operator structs, and `Av1Transform1dMath` | AV1 forward transform definitions and section 7.11.2 inverse transform and reconstruction | libaom `av1/encoder/av1_fwd_txfm2d_hwy.h`, its AVX-512 and AVX2 instantiations, `av1/encoder/av1_fwd_txfm1d.c`, `av1/common/av1_inv_txfm1d_cfg.h`, `av1/common/av1_inv_txfm1d.c`, `av1/common/av1_inv_txfm2d.c`, the x86 AVX2/SSE4 implementations, and the corresponding Neon implementations at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the normative staged DCT, ADST, and identity arithmetic, direction-specific stage ranges and shifts, transposition, clipping, and high-bit-depth sample addition. Stateless static-generic operators follow ImageSharp's JPEG color-transform pattern. Forward operators share one behavior model across scalar, `Vector128`, `Vector256`, and `Vector512`, selecting packed or expanded lane storage at the 2-D boundary. Inverse production traversal retains the verified scalar, `Vector128`, and `Vector256` tiers until a wider upstream shape and complete-block benefit are both established. |
| `Av1ForwardTransformer`, `Av1Inverse2dTransformer`, `Av1Transform2dFlipConfiguration`, their nested forward/inverse 1-D operator structs, and `Av1Transform1dMath` | AV1 forward transform definitions and section 7.11.2 inverse transform and reconstruction | libaom `av1/encoder/av1_fwd_txfm2d_hwy.h`, its AVX-512 and AVX2 instantiations, `av1/encoder/av1_fwd_txfm1d.c`, `av1/common/av1_inv_txfm1d_cfg.h`, `av1/common/av1_inv_txfm1d.c`, `av1/common/av1_inv_txfm2d.c`, the x86 AVX2/SSE4 implementations, and the corresponding Neon implementations at `03087864cf4bea6abb0d28f95cf7843511413d8f` | Preserve the normative staged DCT, ADST, and identity arithmetic, direction-specific stage ranges and shifts, transposition, clipping, and high-bit-depth sample addition. Stateless static-generic operators follow ImageSharp's JPEG color-transform pattern and remain nested in the family that owns their shared traversal. Forward operators share one behavior model across scalar, `Vector128`, `Vector256`, and `Vector512`, selecting packed or expanded lane storage at the 2-D boundary. Inverse production traversal retains the verified scalar, `Vector128`, and `Vector256` tiers until a wider upstream shape and complete-block benefit are both established. |
| `HeifDecoderCore` box extension handling and `HeifDecoderCore`/`HeifEncoderCore` item-property associations | ISO/IEC 14496-12 box extensibility and section 8.11.14 item properties and `ipma` syntax | libavif `src/read.c` and `src/write.c` at `092276ce89098ead06db80975173191e5fee1826` | Skip unrecognized top-level and metadata child boxes, preserve the position of every property in `ipco`, reject an unrecognized property only when its item association marks it essential, associate properties by item ID, and read or write the essential bit plus one-based 7-bit or 15-bit property index according to the full-box flags. Independent HEIC, HIF, and AVIF fixtures provide the reader oracle; container-level identification of encoded output guards the writer independently of pixel roundtripping. |
| `HeifCleanAperture`, `HeifItem` presentation state, and `HeifDecoderCore` transformative-property parsing and application | ISO/IEC 14496-12 section 12.1.4 clean aperture; HEIF image rotation and mirror properties; MIAF section 7.3.6.7 presentation order and section 7.3.9 essential transformative properties | libavif `src/avif.c` clean-aperture conversion, `src/read.c` property parsers and alpha-property validation, and `apps/shared/avifutil.c` transform application at `092276ce89098ead06db80975173191e5fee1826` | Resolve fractional clean-aperture dimensions and center offsets to exact bounded integer pixels, validate the registered rotation/mirror reserved bits, require essential associations, crop after auxiliary-alpha composition, map counter-clockwise HEIF quarter turns to ImageSharp's optimized clockwise rotate modes, then mirror around the signaled axis. Reuse ImageSharp's existing crop, rotation, and flip processors for every pixel type. Retain only the three image-item property values; do not add a generic transform-box or ISO BMFF model. |
| `HeifConstants.IsSupportedFileType`, `HeifImageFormatDetector`, and `HeifDecoderCore.CheckFileTypeBox` | ISO/IEC 14496-12 `FileTypeBox` syntax and the MP4 Registration Authority HEIF/AVIF still-image and sequence brand registrations | libavif `src/read.c` functions `avifParseFileTypeBox`, `avifFileTypeHasBrand`, and `avifFileTypeIsCompatible` at `092276ce89098ead06db80975173191e5fee1826` | Apply one rule to the major and compatible brands, accept implemented still-image brands and the bounded `avis`, `hevc`, and `hevx` image-sequence brands, and distinguish item and sequence presentation before parsing their payloads. Layered HEVC and JPEG sequence brands remain unsupported. The decoder validates the complete `ftyp` payload; the fixed-size format detector inspects the available prefix. |
@ -858,7 +858,7 @@ No valid HEVC or AV1 color, compression, or bit-depth row may remain `unsupporte
- Keep changes vertical and reviewable. A slice should add one behavior, its focused tests, independent evidence, and any required notice update.
- Keep every AV1 prediction family on the established JPEG color-converter operator architecture. Each distinct traversal contract owns a family-named predictor type; its `.Operator.cs` defines the static interface, and its family-named files own the closed generic widest-to-narrowest SIMD traversal. Semantic `readonly struct` operators implement scalar, `Vector128`, `Vector256`, and `Vector512` arithmetic through that contract. Only modes which share the same traversal and contract may share a predictor family; do not nest a separate predictor beneath a broad intra/inter family or create hardware-width-specific class hierarchies.
- Keep AV1 one-dimensional transform implementations as top-level semantic `readonly struct` operators. Forward operators belong to `Transform.Forward`, inverse operators belong to `Transform.Inverse`, and each family implements its top-level static-generic contract. Keep `Av1ForwardTransformer` and `Av1Inverse2dTransformer` as separate two-dimensional dispatchers, and keep `Av1InverseTransformOutputOperator<TSample>` top-level. Do not wrap or nest transform operators beneath either dispatcher and do not name operator files after an implementation-detail owner.
- Keep AV1 one-dimensional transforms on the JPEG color-converter owner pattern. `Av1ForwardTransformer.Operator.cs` defines the nested forward contract, and each `Av1ForwardTransformer.<Semantic>Operator.cs` file defines one nested semantic `readonly struct` with concrete scalar and SIMD bodies. `Av1Inverse2dTransformer` owns the inverse contract and semantic operators in the same layout, while `Av1InverseTransformer` owns its byte and high-bit-depth reconstruction-output operators. Shared lane arithmetic belongs in explicitly named `Operations` types, never in forwarding semantic operators. Do not create top-level transform operators, `Forward1d` or `Inverse1d` wrapper types, generic semantic forwarding cores, CRTP contracts, or hardware-width-named operator types.
- Design SIMD-suitable codec work SIMD-first. Establish vector-friendly storage, operator boundaries, scratch ownership, traversal, every applicable lane width, and benchmark-gated dispatch before implementing the equivalent scalar fallback; never build a scalar production architecture and bolt SIMD onto it later.
- Inspect every owning method and upstream invariant before adding guards. Validate external file data at the parser/model boundary and rely on those established invariants internally.
- Do not extract one-use helpers merely to label code. Extract shared primitives only when they have genuine reuse or remove substantial complexity.

Loading…
Cancel
Save