ImageSharp/HEIF_IMPLEMENTATION_PLAN.md

AVIF and AV1 implementation plan

Goal

Complete a production-quality, fully managed AV1 codec and its bounded AVIF/HEIF image container integration for ImageSharp. The finished work must decode and encode still images and bounded image sequences, preserve source precision, use ImageSharp memory ownership, and provide SIMD-first hot paths with one behaviorally identical scalar fallback.

This plan is the authoritative delivery checklist. A source file, unit test, build, self-roundtrip, or local implementation is not completion evidence by itself.

Source authority

• AV1 codec syntax, tables, fixed-point arithmetic, prediction, transforms, entropy behavior, filters, and lifecycle behavior must be ported and checked only against libaom commit 03087864cf4bea6abb0d28f95cf7843511413d8f.
• The AV1 specification is the normative behavioral description. It does not authorize copying an implementation from another codec library.
• Existing ImageSharp and JPEG code is the architecture, allocator, SIMD dispatch, pixel conversion, and test-API pattern. It is not an alternate AV1 algorithm source.
• Production code must not load, invoke, install, or fall back to a native codec.
• External artifacts may be retained only as test inputs or expected outputs with recorded provenance. They must never become an implementation source.

Status notation

• Verified: the current behavior has exact evidence from the pinned reference and the evidence proves the production contract.
• [~] Locally implemented, checkpoint open: production source exists, but current-tree verification is missing or a known audit issue invalidates the checkpoint.
• Remaining: the production behavior is absent, incomplete, or has not reached its required implementation boundary.

Current source reconciliation

Reconciled with the worktree on 2026-08-30.

• [~] The bounded container reader, still-image path, sequence parser, AV1 decoder, color pipeline, presentation pipeline, and broad AV1 test suite exist locally.
• [~] The inter-frame decoder contains implementations for single-reference prediction, compound references, inter-intra prediction, selectable compound blending, OBMC, scaled references, local warped motion, and global motion. These downstream paths must not be called verified until the single-reference checkpoint below is corrected and rerun.
• [~] Loop filtering, CDEF, super-resolution, restoration, film grain, layered presentation, alpha composition, and color conversion exist locally. Shared-source cleanup changed the current tree, so final production-path verification is open.
• [~] AV1 writer primitives, forward transforms, symbol encoding, and tile-writing source exist locally, but they are not connected to the public encoder.
• The public AV1 encoder is not implemented. HeifEncoderCore.Encode throws NotSupportedException when AV1 is selected.
• Removed codec production code, registrations, tests, benchmarks, fixtures, reference outputs, downloaded tools, downloaded source trees, and notices have been manually deleted and verified by the cleanup evidence below.
• The complete decoder and encoder release matrix is not complete.

Immediate execution queue

Work must proceed in this order. Do not skip to a later item while an earlier checkpoint is open.

1. Finish and verify the AV1-only cleanup

• Remove production types, registrations, constants, parser branches, properties, tests, benchmarks, fixtures, reference outputs, notices, and documentation for removed codec work.
• Remove downloaded non-libaom reference source, tools, generated outputs, and local installations.
• Retain the pinned libaom source and build artifacts required for AV1 verification.
• Retain user-supplied AV1 fixtures and their recorded expected outputs.
• Audit production source, tests, benchmarks, assets, project files, notices, and documentation for stale removed-code references.
• Build the current source targets in Release with restore disabled, build servers disabled, and one MSBuild node.
• Run the focused AV1/container tests needed to prove the cleanup did not damage AVIF behavior.
• Run scoped semantic and StyleCop inspection, whitespace inspection, and git diff --check.
• Record the exact verified evidence in this plan.

Verified cleanup evidence on 2026-08-30:

• Release source builds passed for net10.0 and net11.0 with zero warnings and zero errors. Both builds used --no-restore, --disable-build-servers, and one MSBuild node.
• The focused net10.0 HEIF decoder, encoder, metadata, sequence-parser, and AV1 reconstruction set passed 221 of 221 tests with zero failures and zero skips.
• The Roslyn compiler and configured StyleCop analyzers accepted the changed production source. Roslynk's open_solution entry point was attempted separately but failed before returning a solution handle, so no Roslynk result is claimed.
• The final text and filename audit found no removed-code references outside the unchanged repository and shared-infrastructure .gitattributes patterns.
• git diff --check passed and neither .gitattributes file changed.

2. Correct the single-reference inter-frame checkpoint

The following findings are confirmed by direct source inspection and keep the checkpoint open.

• Correct interpolation-filter syntax in Av1TileReader.
  • Current source treats every global-motion type other than Translation as non-translational.
  • Pinned libaom omits interpolation-filter syntax only when the selected model type is greater than Translation.
  • Identity GLOBALMV blocks of sufficient size must consume switchable-filter symbols.
  • Add production-path syntax coverage using the default Identity model. A test that forces Translation does not prove this rule.
• Correct both spatial reference-MV extension loops in Av1ReferenceMotionVectors.
  • Current source stops spatial extension when the stack reaches two entries.
  • Pinned libaom extends the stack through MAX_REF_MV_STACK_SIZE, which is eight.
  • Preserve DRL candidates and consume every required DRL symbol.
  • Describe this as spatial single-reference extension, never temporal extension.
• Replace the contiguous-span dependency in Av1FrameBuffer and all affected inter reconstruction callers.
  • Current GetPaddedPlaneSpan calls DangerousGetSingleSpan.
  • Buffer2D may use multiple memory groups under a constrained allocator.
  • Implement an efficient group-safe row-oriented contract using established ImageSharp Buffer2D access patterns, or prove and enforce a real contiguous-allocation invariant at the allocator boundary.
  • Do not copy planes and do not allocate per block, row, or scanline.
  • Audit direct DangerousGetSingleSpan use in reconstruction, reference-border extension, film grain, copying, and encoder work rather than fixing only one wrapper.
• Prove the real Av1BlockDecoder.DecodeBlock inter-reconstruction branch.
  • Decode the progressive dependent-frame fixture through the complete public production path.
  • Compare the final frame's native Y, Cb, and Cr planes exactly with pinned libaom output.
  • Compare the final presented image through the established ImageSharp reference-image comparison API.
  • Do not substitute an internal helper test, fake tile reader, non-zero assertion, custom pixel loop, or tolerant comparison.
• Prove motion-field ownership and lifetime.
  • Track initialization, retained-slot aliases, failure unwinding, presentation ownership, decoder-result ownership, and final disposal.
  • Every allocator-owned object must be returned exactly once.
• Correct stale documentation.
  • Av1InterFrameModeInfoTests must describe the behavior it actually proves.
  • Do not claim production reconstruction, constrained allocation, ownership, or reference-stack coverage unless the test executes that contract.

Checkpoint gate:

• Default Identity and Translation GLOBALMV syntax cases pass.
• Eight-entry spatial extension and DRL syntax cases pass.
• The exact dependent-frame native-plane comparison passes.
• The established exact presentation comparison passes.
• Normal, AVX-512-disabled, AVX-disabled, and scalar FeatureTestRunner configurations pass where supported.
• Constrained multi-group allocation passes without copying or per-block allocation.
• Motion-field allocation tracking is balanced across success and failure.
• Release builds for net10.0 and net11.0 pass with zero errors.
• Focused Release tests pass with zero failures or skips.
• Scoped semantic, StyleCop, whitespace, and git diff checks pass.
• Only after all evidence is recorded may this checkpoint be committed.

3. Reverify downstream inter prediction in recorded order

These implementations exist locally but inherit the open single-reference syntax, buffer, and ownership foundation.

• [~] Compound reference selection, paired reference-MV derivation, and equal averaging.
• [~] Inter-intra prediction.
• [~] Distance-weighted compound prediction.
• [~] Wedge compound prediction.
• [~] Difference-weighted compound prediction.
• [~] OBMC.
• [~] Scaled-reference prediction.
• [~] Local warped prediction.
• [~] Non-translational global prediction.
• [~] Inter deblocking decisions and reference/mode deltas.

For every item:

• Trace syntax and arithmetic to the pinned libaom commit.
• Execute the real production decoder path.
• Compare native planes exactly.
• Compare presentation through the established reference-image API.
• Run constrained allocator and exactly-once ownership coverage.
• Run FeatureTestRunner for SIMD and scalar dispatch when the implementation has SIMD.
• Record focused Release evidence before marking the item verified.

4. Close AV1 decoder coverage

Previously verified algorithm checkpoints remain valuable evidence, but the final decoder gate requires a fresh current-tree run after the inter and cleanup corrections.

• Bounded OBU framing, sequence headers, frame headers, tile groups, alignment, and trailing-bit parsing have pinned-reference checkpoint evidence.
• Partition traversal, mode information, segmentation, delta quantization, transform-size selection, coefficient decoding, inverse quantization, and inverse transforms have pinned-reference checkpoint evidence.
• Intra prediction covers directional, DC, smooth, Paeth, chroma-from-luma, filter-intra, and palette families with the established operator architecture.
• Intra-block copy has exact native reconstruction and feature-isolated SIMD evidence.
• Lossless inverse transform, loop filtering, CDEF, super-resolution, restoration, and film grain have focused checkpoint evidence.
• [~] Retained references, CDF snapshots, segmentation maps, global motion, temporal motion fields, and dependent-frame lifecycle exist locally and require current-tree re-verification.
• [~] All-intra and dependent-frame profile fixtures exist for 8, 10, and 12-bit monochrome, 4:2:0, 4:2:2, and 4:4:4 paths.
• Re-run the exact current-tree native-plane matrix through the production decoder.
• Re-run the exact current-tree presentation matrix through ImageSharp's established comparison API.
• Verify malformed/truncated data, frame IDs, reference slots, tile bounds, allocation limits, cancellation, and failure unwinding.
• Verify still items and bounded sequences from file, memory, non-seekable, and short-read streams.
• Verify ICC, CICP, alpha, grids, pixel aspect ratio, clean aperture, rotation, mirroring, metadata, and every presented sequence frame.
• Complete the public AVIF format/API review so registered capabilities match implemented behavior.
• Remove or reject every valid in-scope AV1 syntax branch that remains silently ignored or unsupported.

Decoder exit gate:

• Every supported native format and AV1 tool has exact pinned-libaom production-path evidence.
• Every supported presentation behavior has established reference-image evidence at the correct output precision.
• No decoder path relies on a native codec, copied plane, per-block allocation, or contiguous memory-group accident.
• All allocator ownership is deterministic and exactly once.
• Full focused Release verification is recorded with no false coverage claims.

AV1 encoder implementation

Writer primitives are not an encoder. The public encoder remains incomplete until it produces independently decodable AV1 payloads and AVIF containers for every exposed option.

5. Define and enforce the encoder contract

• Finalize observable options for quality, effort, lossless mode, bit depth, chroma subsampling, alpha quality, metadata, and bounded sequences.
• Preserve high-bit-depth source precision through 16-bit RGB and native 10/12-bit component planes.
• Reject unsupported combinations at the public boundary before writing output.
• Register only capabilities that the completed encoder proves.

6. Build the complete AV1 frame encoder

• [~] SIMD-first RGB-to-native-plane conversion exists locally.
• [~] Forward transform families and transform workspace exist locally.
• [~] Symbol writer, coefficient writer, and tile writer fragments exist locally.
• Connect a frame-owned encoder lifecycle using ImageSharp allocators and pools.
• Write compliant temporal delimiter, sequence header, frame header, tile group, metadata, and padding OBUs as required.
• Implement superblock and partition analysis for every permitted block size and partition.
• Implement intra mode search, chroma mode search, palette, filter intra, chroma-from-luma, and intra-block copy decisions.
• Implement inter mode search for bounded sequences, including reference selection and the decoder-supported inter tools.
• Implement transform-size/type search, forward transform, quantization, coefficient optimization, and lossless behavior.
• Implement real rate-distortion selection and make quality and effort change work, size, and output quality.
• Implement tile-local entropy coding and CDF update behavior.
• Implement legal deblocking, CDEF, restoration, super-resolution, and film-grain signaling decisions.
• Remove per-transform and per-block managed allocations from active encoder paths.
• Use descending SIMD dispatch: Vector512, Vector256, Vector128, then scalar.
• Verify every SIMD operator with FeatureTestRunner and an independent scalar oracle shaped from the same pinned libaom behavior.

7. Write complete AVIF output

• Write the correct AVIF file type, item information, locations, references, properties, AV1 configuration, dimensions, color, alpha, metadata, and media data.
• Support single images, alpha auxiliary images, grids, multiple extents, and bounded image sequences in the final public scope.
• Preserve ICC, Exif, and XMP according to encoder options.
• Write CICP, range, chroma position, bit depth, and subsampling values that match the encoded planes.
• Apply orientation and clean-aperture behavior consistently with ImageSharp encoder conventions.
• Stream output through allocator-backed chunked storage without file-sized copies or ToArray materialization.

Encoder exit gate:

• Pinned libaom accepts every produced AV1 payload.
• Lossless output is exact at native-plane and final-pixel precision.
• Lossy output demonstrates recorded quality and effort tradeoffs with absolute size, quality, timing, and allocation evidence.
• 8, 10, and 12-bit monochrome, 4:2:0, 4:2:2, and 4:4:4 outputs pass.
• Alpha, grids, metadata, color profiles, transforms, and bounded sequences pass.
• ImageSharp decode of its own output is supplemental coverage only, never the sole oracle.
• Public encoding no longer throws for a supported AV1 request.
• Focused Release and FeatureTestRunner verification passes with exact recorded evidence.

Architecture rules

• Follow the JPEG color-converter operator architecture exactly.
• Each distinct prediction traversal owns a family-named predictor type.
• The family .Operator.cs file defines the nested static operator contract.
• Each semantic readonly struct belongs to that owner and implements concrete scalar, Vector128, Vector256, and Vector512 arithmetic for the shared traversal.
• Do not place a distinct predictor beneath a broad Av1IntraPredictor or Av1InterPredictor.
• Do not create semantic forwarding wrappers, top-level operator types, hardware-width-named operator types, CRTP contracts, or one file containing unrelated semantic operators.
• Forward transforms belong to Av1ForwardTransformer and its semantic operator files.
• Inverse axis transforms belong to Av1Inverse2dTransformer and its semantic operator files.
• Reconstruction output operators belong to Av1InverseTransformer.
• Shared lane primitives belong only in explicitly named Operations types.
• Dispatch from widest to narrowest supported SIMD width, then execute one scalar tail.
• Keep codec execution sequential. Do not add parallel execution inside the codec.
• Do not allocate per row, block, transform, scanline, or SIMD tail.
• Use ImageSharp allocators and pools. Do not use ToArray to cross an ownership boundary.
• On internal types, use public members when other types consume them; reserve private members for type-local behavior.
• Use established ImageSharp test data, allocator tracking, FeatureTestRunner, and reference-image comparison APIs. Do not build custom substitutes.
• Public XML documentation describes observable behavior only.
• Inline comments explain the pinned numerical rule, ownership boundary, edge extension, entropy ordering, or SIMD shape at technically complex points.
• Every multiline statement or declaration is followed by vertical whitespace.
• Do not edit .gitattributes directly.
• Do not install or download tools without explicit permission.

Final verification matrix

• Release source build: net10.0, zero errors.
• Release source build: net11.0, zero errors.
• Scoped semantic inspection: zero compiler errors attributable to this work.
• Focused decoder syntax, reconstruction, ownership, presentation, and malformed-input tests.
• Focused encoder syntax, payload, container, precision, ownership, and option tests.
• FeatureTestRunner coverage for normal, narrower SIMD tiers, and scalar fallback.
• Constrained multi-group allocator coverage with balanced exactly-once returns.
• Exact native-plane comparisons against pinned libaom.
• Established final-presentation comparisons at the target pixel precision.
• Scoped StyleCop and vertical-whitespace inspection.
• No stale unsupported capability claims or removed-code references.
• No restore-source failures, background test hosts, detached processes, or crash-report popups.
• .gitattributes unchanged.
• git diff --check clean.
• Documentation records exact commands, counts, fixture hashes, and results.
• Commit only after the relevant checkpoint is genuinely complete.
• Do not push.