Decode HEIF image grids

HEIF_IMPLEMENTATION_PLAN.md

@@ -77,6 +77,7 @@ This snapshot pins or classifies the available references and failures; it does
 | `HeifConstants.IsSupportedFileType`, `HeifImageFormatDetector`, and `HeifDecoderCore.CheckFileTypeBox` | ISO/IEC 14496-12 `FileTypeBox` syntax and the MP4 Registration Authority HEIF/AVIF still-image and sequence brand registrations | libavif `src/read.c` functions `avifParseFileTypeBox`, `avifFileTypeHasBrand`, and `avifFileTypeIsCompatible` at `092276ce89098ead06db80975173191e5fee1826` | Apply one rule to the major and compatible brands, accept the implemented still-image container and payload brands, and reject registered HEVC, AVIF, and JPEG sequence major brands as outside the image-item scope. The decoder validates the complete `ftyp` payload; the fixed-size format detector inspects the available prefix. |
 | `HeifDecoderCore.ReadBoxHeader` and `HeifDecoderCore.ParseBoxHeader` | ISO/IEC 14496-12 section 4.2.2 basic box syntax | libavif `src/stream.c` functions `avifROStreamReadBoxHeaderPartial` and `avifROStreamReadBoxHeader` at `092276ce89098ead06db80975173191e5fee1826` | Resolve 32-bit, 64-bit, UUID, and top-level size-zero boxes into content lengths only after validating the complete variable-sized header and the remaining parent boundary. Nested size-zero boxes are invalid; large skips retain 64-bit offsets. |
 | `HeifDecoderCore.ParseMetadata` | ISO/IEC 14496-12 `MetaBox` and HEIF item declarations, locations, properties, and associations | libavif `src/read.c` functions `avifParseMetaBox`, `avifMetaFindOrCreateItem`, `avifParseItemLocationBox`, and `avifParseItemPropertiesBox` at `092276ce89098ead06db80975173191e5fee1826` | Index unique recognized metadata children by type and payload location, then parse them in dependency order so physical placement does not control item lookup or property association. Duplicate unique children and truncated full-box headers are invalid. |
+| `GridHeifItemDecoder` and `HeifDecoderCore` grid/thumbnail selection | ISO/IEC 23008-12 section 6.6.2.3 image-grid syntax and MIAF grid-cell constraints | libavif `src/read.c` functions `avifParseImageGridBox`, `avifDecoderDataAllocateImagePlanes`, and `avifDecoderDataCopyTileToImage` at `092276ce89098ead06db80975173191e5fee1826` | Parse version-zero 16-bit and 32-bit grid descriptors, preserve row-major `dimg` order, require the declared tile count and one coding format, validate canvas coverage and edge overlap, and crop only the rightmost column and bottom row while copying through ImageSharp pixel buffers. A primary grid whose tile codec is unavailable may use only a decodable thumbnail that explicitly references that grid. |
 
 This table is intentionally incomplete. Add a row before each additional AV1 or HEVC algorithm is ported or materially reshaped.
 
@@ -100,7 +101,7 @@ This assessment is based on the current source after the upstream ImageSharp mer
 - Item IDs are resolved as keys rather than list indices; metadata children are indexed and parsed in dependency order rather than physical order.
 - Item property associations now preserve physical `ipco` indices, apply one-based 7-bit or 15-bit indices and essential flags, associate by item ID, and reject arbitrary unknown essential properties.
 - Item locations now support bounded file-relative and `idat`-relative storage, multiple ordered extents, versioned item IDs, 0/4/8-byte registered field sizes, and 64-bit offsets. Referenced-item construction method two and external data references remain explicitly unsupported.
-- Grid decoder selection is no longer overwritten and `dimg` references are selected explicitly, but `GridHeifItemDecoder` still returns a placeholder 1x1 image instead of parsing the grid descriptor and composing validated tiles.
+- Grid derived-image decoding now parses both registered descriptor widths, resolves the ordered `dimg` cells, validates tile count, coding format, dimensions, canvas coverage, and edge overlap, then composes the output through ImageSharp row buffers. Unsupported grid tile codecs can select only a decodable thumbnail of the same primary grid. Independent AV1 and JPEG grid fixtures are still required, and HEVC grids remain blocked on the HEVC decoder.
 - HEVC and AV1 configuration, CICP color information, alpha auxiliary items, Exif/XMP, transforms, and several item/property relationships are missing or parsed without affecting output.
 - Identify and decode now use the same bounded metadata parser and both validate the complete leading file type box. The parsed state is still mutable and Identify does not yet report the complete bit depth, color, alpha, profile, or transform model.
 

src/ImageSharp/Formats/Heif/GridHeifItemDecoder.cs

@@ -2,6 +2,7 @@
 // Licensed under the Six Labors Split License.
 
 using System.Buffers;
+using System.Buffers.Binary;
 using SixLabors.ImageSharp.Common.Helpers;
 using SixLabors.ImageSharp.Memory;
 using SixLabors.ImageSharp.PixelFormats;
@@ -69,33 +70,134 @@ internal class GridHeifItemDecoder<TPixel> : IHeifItemDecoder<TPixel>
     /// <returns>The image reconstructed from the referenced grid tiles.</returns>
     public Image<TPixel> DecodeItemData(Configuration configuration, HeifItem gridItem, Span<byte> data)
     {
-        List<uint> linked = this.itemLinks.First(
+        if (data.Length < 8)
-            link => link.Type == Heif4CharCode.Dimg && link.SourceId == gridItem.Id).DestinationIds;
+        {
+            throw new InvalidImageContentException("The HEIF image grid descriptor is truncated.");
+        }
+
+        byte version = data[0];
+        if (version != 0)
+        {
+            throw new InvalidImageContentException($"The HEIF image grid descriptor has unsupported version {version}.");
+        }
+
+        byte flags = data[1];
+        int rows = data[2] + 1;
+        int columns = data[3] + 1;
+        bool usesLargeDimensions = (flags & 1) != 0;
+        int descriptorLength = usesLargeDimensions ? 12 : 8;
+        if (data.Length != descriptorLength)
+        {
+            throw new InvalidImageContentException("The HEIF image grid descriptor has an invalid length.");
+        }
+
+        uint outputWidth;
+        uint outputHeight;
+        if (usesLargeDimensions)
+        {
+            outputWidth = BinaryPrimitives.ReadUInt32BigEndian(data[4..]);
+            outputHeight = BinaryPrimitives.ReadUInt32BigEndian(data[8..]);
+        }
+        else
+        {
+            outputWidth = BinaryPrimitives.ReadUInt16BigEndian(data[4..]);
+            outputHeight = BinaryPrimitives.ReadUInt16BigEndian(data[6..]);
+        }
+
+        if (outputWidth is 0 or > int.MaxValue || outputHeight is 0 or > int.MaxValue)
+        {
+            throw new InvalidImageContentException("The HEIF image grid descriptor has invalid output dimensions.");
+        }
+
+        List<uint> linked = [];
+        foreach (HeifItemLink link in this.itemLinks)
+        {
+            if (link.Type == Heif4CharCode.Dimg && link.SourceId == gridItem.Id)
+            {
+                // The order of dimg destinations is the normative row-major order of the grid cells.
+                linked.AddRange(link.DestinationIds);
+            }
+        }
+
+        int tileCount = rows * columns;
+        if (linked.Count != tileCount)
+        {
+            string message = $"The HEIF image grid requires {tileCount} tiles, but its derived-image references contain {linked.Count}.";
+            throw new InvalidImageContentException(message);
+        }
 
         // Each compressed tile decoder returns an owned Image. Keep every tile alive until
         // the final grid has copied its pixels, then dispose all intermediates together.
         using DisposableList<Image<TPixel>> gridTiles = new(linked.Count);
+        Heif4CharCode tileType = default;
         foreach (uint id in linked)
         {
-            HeifItem? item = this.items.FirstOrDefault(item => item.Id == id);
+            HeifItem item = this.items.First(item => item.Id == id);
-            if (item is not null)
+            if (tileType == default)
+            {
+                tileType = item.Type;
+            }
+            else if (item.Type != tileType)
+            {
+                throw new InvalidImageContentException("All HEIF image grid tiles must use the same coding format.");
+            }
+
+            IHeifItemDecoder<TPixel>? decoder = HeifCompressionFactory.GetDecoder<TPixel>(item.Type);
+            if (decoder is null)
             {
-                IHeifItemDecoder<TPixel>? decoder = HeifCompressionFactory.GetDecoder<TPixel>(item.Type);
+                throw new ImageFormatException($"The HEIF image grid uses unsupported tile type '{item.Type}'.");
-                if (decoder is not null)
-                {
-                    this.CompressionMethod = decoder.CompressionMethod;
-                    IMemoryOwner<byte> itemMemory = this.buffers[item.Id];
-                    gridTiles.Add(decoder.DecodeItemData(this.configuration, item, itemMemory.GetSpan()));
-                }
             }
+
+            if (!this.buffers.TryGetValue(item.Id, out IMemoryOwner<byte>? itemMemory))
+            {
+                throw new InvalidImageContentException($"HEIF image grid tile {item.Id} has no data extents.");
+            }
+
+            this.CompressionMethod = decoder.CompressionMethod;
+            gridTiles.Add(decoder.DecodeItemData(this.configuration, item, itemMemory.GetSpan()));
+        }
+
+        Image<TPixel> firstTile = gridTiles[0];
+        int tileWidth = firstTile.Width;
+        int tileHeight = firstTile.Height;
+        if (((long)tileWidth * columns) < outputWidth || ((long)tileHeight * rows) < outputHeight)
+        {
+            throw new InvalidImageContentException("The HEIF image grid tiles do not cover the output canvas.");
+        }
+
+        if (((long)tileWidth * (columns - 1)) >= outputWidth || ((long)tileHeight * (rows - 1)) >= outputHeight)
+        {
+            throw new InvalidImageContentException("The HEIF image grid edge tiles do not overlap the output canvas.");
         }
 
-        if (gridTiles.Count == 0)
+        Image<TPixel> result = new(configuration, (int)outputWidth, (int)outputHeight, firstTile.Metadata.DeepClone());
+        ImageFrame<TPixel> destination = result.Frames.RootFrame;
+        for (int tileIndex = 0; tileIndex < gridTiles.Count; tileIndex++)
         {
-            return new Image<TPixel>(1, 1);
+            Image<TPixel> tile = gridTiles[tileIndex];
+            if (tile.Width != tileWidth || tile.Height != tileHeight)
+            {
+                result.Dispose();
+                throw new InvalidImageContentException("The HEIF image grid contains tiles with mismatched dimensions.");
+            }
+
+            int column = tileIndex % columns;
+            int row = tileIndex / columns;
+            int destinationX = column * tileWidth;
+            int destinationY = row * tileHeight;
+            int copyWidth = Math.Min(tileWidth, (int)outputWidth - destinationX);
+            int copyHeight = Math.Min(tileHeight, (int)outputHeight - destinationY);
+            ImageFrame<TPixel> source = tile.Frames.RootFrame;
+
+            // The descriptor may crop only the rightmost column and bottom row. Copying bounded row spans
+            // applies that crop without allocating derived-image views or invoking the processing pipeline.
+            for (int y = 0; y < copyHeight; y++)
+            {
+                Span<TPixel> destinationRow = destination.PixelBuffer.DangerousGetRowSpan(destinationY + y).Slice(destinationX, copyWidth);
+                source.PixelBuffer.DangerousGetRowSpan(y)[..copyWidth].CopyTo(destinationRow);
+            }
         }
 
-        // TODO: Combine grid tiles into a single image.
+        return result;
-        return new Image<TPixel>(1, 1);
     }
 }

src/ImageSharp/Formats/Heif/HeifDecoderCore.cs

@@ -207,36 +207,11 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
         {
             // A grid is a derived image rather than a compression method. Its dimg references identify the coded
             // tile items whose decoder determines the compression reported for the primary presentation.
-            HeifItemLink? derivedImageReference = this.itemLinks.FirstOrDefault(
+            metadataItem = this.FindDecodableGridTile<Rgba32>(item) ?? this.FindDecodableThumbnail<Rgba32>(item) ?? item;
-                link => link.Type == Heif4CharCode.Dimg && link.SourceId == item.Id);
-
-            if (derivedImageReference is not null)
-            {
-                HeifItem? tileItem = derivedImageReference.DestinationIds
-                    .Select(this.FindItemById)
-                    .FirstOrDefault(candidate => candidate is not null && HeifCompressionFactory.GetDecoder<Rgba32>(candidate.Type) is not null);
-
-                if (tileItem is not null)
-                {
-                    metadataItem = tileItem;
-                }
-            }
         }
         else if (HeifCompressionFactory.GetDecoder<Rgba32>(item.Type) is null)
         {
-            // A thumbnail reference points from the thumbnail item to the master image. Restrict fallback metadata
+            metadataItem = this.FindDecodableThumbnail<Rgba32>(item) ?? item;
-            // to a thumbnail of this primary item rather than allowing an unrelated thumbnail to relabel it.
-            HeifItemLink? thumbnailReference = this.itemLinks.FirstOrDefault(
-                link => link.Type == Heif4CharCode.Thmb && link.DestinationIds.Contains(item.Id));
-
-            if (thumbnailReference is not null)
-            {
-                HeifItem? thumbnailItem = this.FindItemById(thumbnailReference.SourceId);
-                if (thumbnailItem is not null && HeifCompressionFactory.GetDecoder<Rgba32>(thumbnailItem.Type) is not null)
-                {
-                    metadataItem = thumbnailItem;
-                }
-            }
         }
 
         HeifMetadata meta = metadata.GetHeifMetadata();
@@ -1221,26 +1196,19 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
         }
 
         IHeifItemDecoder<TPixel>? itemDecoder = rootItem.Type == Heif4CharCode.Grid
-            ? new GridHeifItemDecoder<TPixel>(this.configuration, this.items, this.itemLinks, buffers)
+            && this.FindDecodableGridTile<TPixel>(rootItem) is not null
-            : HeifCompressionFactory.GetDecoder<TPixel>(rootItem.Type);
+                ? new GridHeifItemDecoder<TPixel>(this.configuration, this.items, this.itemLinks, buffers)
+                : HeifCompressionFactory.GetDecoder<TPixel>(rootItem.Type);
+
         HeifItem itemToDecode = rootItem;
         if (itemDecoder is null)
         {
             // Unable to decode the primary image, decode the thumbnail instead.
-            HeifItemLink? thumbLink = this.itemLinks.FirstOrDefault(
+            HeifItem? thumbnailItem = this.FindDecodableThumbnail<TPixel>(rootItem);
-                link => link.Type == Heif4CharCode.Thmb && link.DestinationIds.Contains(rootItem.Id));
+            if (thumbnailItem is not null)
-
-            if (thumbLink is not null)
             {
-                HeifItem? thumbItem = this.FindItemById(thumbLink.SourceId);
+                itemDecoder = HeifCompressionFactory.GetDecoder<TPixel>(thumbnailItem.Type);
-                if (thumbItem is not null)
+                itemToDecode = thumbnailItem;
-                {
-                    itemDecoder = HeifCompressionFactory.GetDecoder<TPixel>(thumbItem.Type);
-                    if (itemDecoder is not null)
-                    {
-                        itemToDecode = thumbItem;
-                    }
-                }
             }
         }
 
@@ -1339,6 +1307,68 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
     private HeifItem? FindItemById(uint itemId)
         => this.items.FirstOrDefault(item => item.Id == itemId);
 
+    /// <summary>
+    /// Finds the first tile of a grid when every referenced tile uses a registered still-image decoder.
+    /// </summary>
+    /// <typeparam name="TPixel">The destination pixel format used to select item decoders.</typeparam>
+    /// <param name="gridItem">The grid derived-image item.</param>
+    /// <returns>The first decodable grid tile, or <see langword="null"/> when the grid has no tiles or any tile cannot be decoded.</returns>
+    private HeifItem? FindDecodableGridTile<TPixel>(HeifItem gridItem)
+        where TPixel : unmanaged, IPixel<TPixel>
+    {
+        HeifItem? firstTile = null;
+        foreach (HeifItemLink link in this.itemLinks)
+        {
+            if (link.Type != Heif4CharCode.Dimg || link.SourceId != gridItem.Id)
+            {
+                continue;
+            }
+
+            foreach (uint itemId in link.DestinationIds)
+            {
+                HeifItem tile = this.FindItemById(itemId)!;
+                if (HeifCompressionFactory.GetDecoder<TPixel>(tile.Type) is null)
+                {
+                    // A partially decodable grid cannot yield the requested canvas. Returning no tile lets the
+                    // caller select a thumbnail of the complete primary presentation when one is available.
+                    return null;
+                }
+
+                firstTile ??= tile;
+            }
+        }
+
+        return firstTile;
+    }
+
+    /// <summary>
+    /// Finds a decodable thumbnail that represents the specified master image item.
+    /// </summary>
+    /// <typeparam name="TPixel">The destination pixel format used to select item decoders.</typeparam>
+    /// <param name="masterItem">The master image item referenced by the thumbnail.</param>
+    /// <returns>A decodable thumbnail item, or <see langword="null"/> when no matching thumbnail is available.</returns>
+    private HeifItem? FindDecodableThumbnail<TPixel>(HeifItem masterItem)
+        where TPixel : unmanaged, IPixel<TPixel>
+    {
+        // A thumbnail reference points from the thumbnail item to the master image. Restrict fallback to this
+        // presentation rather than allowing an unrelated thumbnail elsewhere in the file to be selected.
+        HeifItemLink? thumbnailReference = this.itemLinks.FirstOrDefault(
+            link => link.Type == Heif4CharCode.Thmb && link.DestinationIds.Contains(masterItem.Id));
+
+        if (thumbnailReference is null)
+        {
+            return null;
+        }
+
+        HeifItem thumbnailItem = this.FindItemById(thumbnailReference.SourceId)!;
+        if (HeifCompressionFactory.GetDecoder<TPixel>(thumbnailItem.Type) is null)
+        {
+            return null;
+        }
+
+        return thumbnailItem;
+    }
+
     /// <summary>
     /// Decodes the UTF-8 bytes preceding the first null terminator.
     /// </summary>