Browse Source

Decode HEIF alpha auxiliary images

pull/2633/head
James Jackson-South 2 weeks ago
parent
commit
77cc468752
  1. 10
      HEIF_IMPLEMENTATION_PLAN.md
  2. 41
      src/ImageSharp/Formats/Heif/GridHeifItemDecoder.cs
  3. 15
      src/ImageSharp/Formats/Heif/Heif4CharCode.cs
  4. 3
      src/ImageSharp/Formats/Heif/Heif4CharCode.tt
  5. 18
      src/ImageSharp/Formats/Heif/HeifConstants.cs
  6. 2
      src/ImageSharp/Formats/Heif/HeifDecoder.cs
  7. 306
      src/ImageSharp/Formats/Heif/HeifDecoderCore.cs
  8. 5
      src/ImageSharp/Formats/Heif/HeifItem.cs
  9. 33
      src/ImageSharp/Formats/Heif/HeifItemDecoderUtilities.cs
  10. 23
      src/ImageSharp/Formats/Heif/HeifMetadata.cs

10
HEIF_IMPLEMENTATION_PLAN.md

@ -78,6 +78,7 @@ This snapshot pins or classifies the available references and failures; it does
| `HeifDecoderCore.ReadBoxHeader` and `HeifDecoderCore.ParseBoxHeader` | ISO/IEC 14496-12 section 4.2.2 basic box syntax | libavif `src/stream.c` functions `avifROStreamReadBoxHeaderPartial` and `avifROStreamReadBoxHeader` at `092276ce89098ead06db80975173191e5fee1826` | Resolve 32-bit, 64-bit, UUID, and top-level size-zero boxes into content lengths only after validating the complete variable-sized header and the remaining parent boundary. Nested size-zero boxes are invalid; large skips retain 64-bit offsets. |
| `HeifDecoderCore.ParseMetadata` | ISO/IEC 14496-12 `MetaBox` and HEIF item declarations, locations, properties, and associations | libavif `src/read.c` functions `avifParseMetaBox`, `avifMetaFindOrCreateItem`, `avifParseItemLocationBox`, and `avifParseItemPropertiesBox` at `092276ce89098ead06db80975173191e5fee1826` | Index unique recognized metadata children by type and payload location, then parse them in dependency order so physical placement does not control item lookup or property association. Duplicate unique children and truncated full-box headers are invalid. |
| `GridHeifItemDecoder` and `HeifDecoderCore` grid/thumbnail selection | ISO/IEC 23008-12 section 6.6.2.3 image-grid syntax and MIAF grid-cell constraints | libavif `src/read.c` functions `avifParseImageGridBox`, `avifDecoderDataAllocateImagePlanes`, and `avifDecoderDataCopyTileToImage` at `092276ce89098ead06db80975173191e5fee1826` | Parse version-zero 16-bit and 32-bit grid descriptors, preserve row-major `dimg` order, require the declared tile count and one coding format, validate canvas coverage and edge overlap, and crop only the rightmost column and bottom row while copying through ImageSharp pixel buffers. A primary grid whose tile codec is unavailable may use only a decodable thumbnail that explicitly references that grid. |
| `HeifDecoderCore` alpha auxiliary selection/composition and `GridHeifItemDecoder` auxiliary tile ordering | ISO/IEC 23008-12 alpha auxiliary image semantics, `auxC`, `auxl`, `prem`, and per-grid-tile alpha relationships | libavif `src/read.c` functions `avifParseAuxiliaryTypeProperty`, `avifDecoderItemIsAlphaAux`, `avifMetaFindAlphaItem`, and `avifDecoderCheckAlphaProperties`, plus `src/scale.c` box-filter scaling at `092276ce89098ead06db80975173191e5fee1826` | Recognize both registered alpha URNs, decode a direct alpha image/grid or the complete row-major set of per-color-tile alpha auxiliaries, normalize through `L16`, box-resample differing auxiliary dimensions, compose through `Rgba64` and `PixelOperations<TPixel>`, and unassociate `prem` color samples with transparent-black handling. This remains an image-item relationship only; no track or generic media-reference model is introduced. |
This table is intentionally incomplete. Add a row before each additional AV1 or HEVC algorithm is ported or materially reshaped.
@ -88,8 +89,8 @@ This assessment is based on the current source after the upstream ImageSharp mer
### Public integration
- `HeifFormat` combines the HEIF, HEIC, HIF, and AVIF identities and extensions, but the implementation does not yet decode all payloads that contract implies.
- `HeifDecoder` defaults to `Rgb24`, which cannot preserve decoded alpha.
- `HeifMetadata` reports fixed 8-bit, three-component RGB metadata rather than the decoded HEVC/AV1 and item properties.
- `HeifDecoder` now defaults to `Rgba32`, preserving decoded auxiliary alpha for non-generic loads.
- `HeifMetadata` now reports alpha presence and the corresponding 24/32-bit RGB pixel shape, but complete decoded HEVC/AV1 bit depth, monochrome/chroma layout, color signaling, and profiles remain absent.
- `IHeifEncoderOptions` is empty, and the encoder exposes no meaningful quality, speed, lossless, subsampling, bit-depth, or alpha policy.
- HEIF/HEIC/AVIF is absent from the format source-generation list in `_Formats.ttinclude`, so the standard ImageSharp save extensions are not generated.
- Configuration registration exists, but it currently registers capabilities broader than the implementation provides.
@ -102,8 +103,9 @@ This assessment is based on the current source after the upstream ImageSharp mer
- Item property associations now preserve physical `ipco` indices, apply one-based 7-bit or 15-bit indices and essential flags, associate by item ID, and reject arbitrary unknown essential properties.
- Item locations now support bounded file-relative and `idat`-relative storage, multiple ordered extents, versioned item IDs, 0/4/8-byte registered field sizes, and 64-bit offsets. Referenced-item construction method two and external data references remain explicitly unsupported.
- Grid derived-image decoding now parses both registered descriptor widths, resolves the ordered `dimg` cells, validates tile count, coding format, dimensions, canvas coverage, and edge overlap, then composes the output through ImageSharp row buffers. Unsupported grid tile codecs can select only a decodable thumbnail of the same primary grid. Independent AV1 and JPEG grid fixtures are still required, and HEVC grids remain blocked on the HEVC decoder.
- HEVC and AV1 configuration, CICP color information, alpha auxiliary items, Exif/XMP, transforms, and several item/property relationships are missing or parsed without affecting output.
- Identify and decode now use the same bounded metadata parser and both validate the complete leading file type box. The parsed state is still mutable and Identify does not yet report the complete bit depth, color, alpha, profile, or transform model.
- Alpha auxiliary decoding now recognizes `auxC`, `auxl`, and `prem`, supports a direct auxiliary image/grid and the per-color-grid-tile form, preserves normalized alpha through `L16`/`Rgba64`, box-resamples differing plane sizes, and reports alpha presence. HEVC alpha remains blocked on the HEVC image-item decoder, while independent AVIF alpha fixtures are still required against the incomplete AV1 reconstruction pipeline.
- HEVC and AV1 configuration, CICP color information, Exif/XMP, transforms, and several image-item properties/relationships are missing or parsed without affecting output.
- Identify and decode now use the same bounded metadata parser and both validate the complete leading file type box. The parsed state is still mutable and Identify does not yet report the complete bit depth, color, profile, or transform model.
### HEVC decoder and encoder

41
src/ImageSharp/Formats/Heif/GridHeifItemDecoder.cs

@ -36,6 +36,11 @@ internal class GridHeifItemDecoder<TPixel> : IHeifItemDecoder<TPixel>
/// </summary>
private readonly IDictionary<uint, IMemoryOwner<byte>> buffers;
/// <summary>
/// The optional row-major tile identifiers supplied for an auxiliary grid plane.
/// </summary>
private readonly IReadOnlyList<uint>? tileItemIds;
/// <summary>
/// Initializes a new instance of the <see cref="GridHeifItemDecoder{TPixel}"/> class.
/// </summary>
@ -43,12 +48,21 @@ internal class GridHeifItemDecoder<TPixel> : IHeifItemDecoder<TPixel>
/// <param name="items">The item definitions in the containing HEIF file.</param>
/// <param name="itemLinks">The item-reference relationships in the containing HEIF file.</param>
/// <param name="buffers">The assembled encoded payload for each image item.</param>
public GridHeifItemDecoder(Configuration configuration, IList<HeifItem> items, IList<HeifItemLink> itemLinks, IDictionary<uint, IMemoryOwner<byte>> buffers)
/// <param name="tileItemIds">
/// Optional row-major tile identifiers that replace the grid item's own derived-image references.
/// </param>
public GridHeifItemDecoder(
Configuration configuration,
IList<HeifItem> items,
IList<HeifItemLink> itemLinks,
IDictionary<uint, IMemoryOwner<byte>> buffers,
IReadOnlyList<uint>? tileItemIds = null)
{
this.configuration = configuration;
this.items = items;
this.itemLinks = itemLinks;
this.buffers = buffers;
this.tileItemIds = tileItemIds;
}
/// <summary>
@ -109,13 +123,16 @@ internal class GridHeifItemDecoder<TPixel> : IHeifItemDecoder<TPixel>
throw new InvalidImageContentException("The HEIF image grid descriptor has invalid output dimensions.");
}
List<uint> linked = [];
foreach (HeifItemLink link in this.itemLinks)
List<uint> linked = this.tileItemIds is null ? [] : new(this.tileItemIds);
if (this.tileItemIds is null)
{
if (link.Type == Heif4CharCode.Dimg && link.SourceId == gridItem.Id)
foreach (HeifItemLink link in this.itemLinks)
{
// The order of dimg destinations is the normative row-major order of the grid cells.
linked.AddRange(link.DestinationIds);
if (link.Type == Heif4CharCode.Dimg && link.SourceId == gridItem.Id)
{
// The order of dimg destinations is the normative row-major order of the grid cells.
linked.AddRange(link.DestinationIds);
}
}
}
@ -154,7 +171,17 @@ internal class GridHeifItemDecoder<TPixel> : IHeifItemDecoder<TPixel>
}
this.CompressionMethod = decoder.CompressionMethod;
gridTiles.Add(decoder.DecodeItemData(this.configuration, item, itemMemory.GetSpan()));
Image<TPixel> tile = decoder.DecodeItemData(this.configuration, item, itemMemory.GetSpan());
try
{
HeifItemDecoderUtilities.ScaleToItemExtent(tile, item);
gridTiles.Add(tile);
}
catch
{
tile.Dispose();
throw;
}
}
Image<TPixel> firstTile = gridTiles[0];

15
src/ImageSharp/Formats/Heif/Heif4CharCode.cs

@ -108,6 +108,11 @@ public enum Heif4CharCode : uint
/// </summary>
Pixi = 0x70697869U,
/// <summary>
/// Auxiliary Type.
/// </summary>
AuxC = 0x61757843U,
/// <summary>
/// Reference Location.
/// </summary>
@ -248,6 +253,16 @@ public enum Heif4CharCode : uint
/// </summary>
Thmb = 0x74686D62U,
/// <summary>
/// Auxiliary Image.
/// </summary>
Auxl = 0x6175786CU,
/// <summary>
/// Premultiplied Alpha.
/// </summary>
Prem = 0x7072656DU,
/// <summary>
/// Content Description.
/// </summary>

3
src/ImageSharp/Formats/Heif/Heif4CharCode.tt

@ -26,6 +26,7 @@
"iscl", "Image Scaling",
"pasp", "Pixel Aspect Ratio",
"pixi", "Pixel Information",
"auxC", "Auxiliary Type",
"rloc", "Reference Location",
"udes", "User Description",
"ipmc", "IPMP Control Box",
@ -54,6 +55,8 @@
"grid", "Grid",
"dimg", "Derived Image",
"thmb", "Thumbnail",
"auxl", "Auxiliary Image",
"prem", "Premultiplied Alpha",
"cdsc", "Content Description",
"mime", "MIME type",
"uri ", "URI",

18
src/ImageSharp/Formats/Heif/HeifConstants.cs

@ -15,6 +15,16 @@ internal static class HeifConstants
/// </summary>
public const Heif4CharCode HeicBrand = Heif4CharCode.Heic;
/// <summary>
/// The auxiliary-type URN used by current HEIF alpha image items.
/// </summary>
public const string AlphaAuxiliaryType = "urn:mpeg:mpegB:cicp:systems:auxiliary:alpha";
/// <summary>
/// The auxiliary-type URN used by legacy HEVC alpha image items.
/// </summary>
public const string LegacyAlphaAuxiliaryType = "urn:mpeg:hevc:2015:auxid:1";
/// <summary>
/// The list of mimetypes that equate to a HEIC.
/// </summary>
@ -69,6 +79,14 @@ internal static class HeifConstants
return false;
}
/// <summary>
/// Determines whether an auxiliary-type property identifies an alpha image plane.
/// </summary>
/// <param name="auxiliaryType">The null-terminated auxiliary type decoded from an <c>auxC</c> property.</param>
/// <returns><see langword="true"/> when the type is either registered HEIF alpha URN.</returns>
public static bool IsAlphaAuxiliaryType(string? auxiliaryType)
=> auxiliaryType is AlphaAuxiliaryType or LegacyAlphaAuxiliaryType;
/// <summary>
/// Determines whether <paramref name="brand"/> identifies a still-image container supported by this codec.
/// </summary>

2
src/ImageSharp/Formats/Heif/HeifDecoder.cs

@ -45,5 +45,5 @@ public sealed class HeifDecoder : ImageDecoder
/// <inheritdoc />
protected override Image Decode(DecoderOptions options, Stream stream, CancellationToken cancellationToken)
=> this.Decode<Rgb24>(options, stream, cancellationToken);
=> this.Decode<Rgba32>(options, stream, cancellationToken);
}

306
src/ImageSharp/Formats/Heif/HeifDecoderCore.cs

@ -11,6 +11,7 @@ using SixLabors.ImageSharp.Memory;
using SixLabors.ImageSharp.Metadata;
using SixLabors.ImageSharp.Metadata.Profiles.Icc;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
namespace SixLabors.ImageSharp.Formats.Heif;
@ -202,16 +203,22 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
/// <param name="item">The primary item whose visible representation is being identified.</param>
private void UpdateMetadata(ImageMetadata metadata, HeifItem item)
{
HeifItem presentationItem = item;
HeifItem metadataItem = item;
if (item.Type == Heif4CharCode.Grid)
{
// A grid is a derived image rather than a compression method. Its dimg references identify the coded
// tile items whose decoder determines the compression reported for the primary presentation.
metadataItem = this.FindDecodableGridTile<Rgba32>(item) ?? this.FindDecodableThumbnail<Rgba32>(item) ?? item;
HeifItem? gridTile = this.FindDecodableGridTile<Rgba32>(item);
HeifItem? thumbnail = gridTile is null ? this.FindDecodableThumbnail<Rgba32>(item) : null;
metadataItem = gridTile ?? thumbnail ?? item;
presentationItem = thumbnail ?? item;
}
else if (HeifCompressionFactory.GetDecoder<Rgba32>(item.Type) is null)
{
metadataItem = this.FindDecodableThumbnail<Rgba32>(item) ?? item;
HeifItem? thumbnail = this.FindDecodableThumbnail<Rgba32>(item);
metadataItem = thumbnail ?? item;
presentationItem = thumbnail ?? item;
}
HeifMetadata meta = metadata.GetHeifMetadata();
@ -226,6 +233,8 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
}
meta.CompressionMethod = compressionMethod;
meta.HasAlpha = this.FindAlphaItem(presentationItem) is not null
|| (presentationItem.Type == Heif4CharCode.Grid && this.FindGridAlphaTiles(presentationItem) is not null);
}
/// <summary>
@ -772,9 +781,14 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
EnsureBufferRemaining(boxBuffer, 0, 12, "image spatial extents");
// The full-box header precedes the unsigned display width and height.
int width = (int)BinaryPrimitives.ReadUInt32BigEndian(boxBuffer[4..]);
int height = (int)BinaryPrimitives.ReadUInt32BigEndian(boxBuffer[8..]);
properties.Add(new KeyValuePair<Heif4CharCode, object>(Heif4CharCode.Ispe, new Size(width, height)));
uint width = BinaryPrimitives.ReadUInt32BigEndian(boxBuffer[4..]);
uint height = BinaryPrimitives.ReadUInt32BigEndian(boxBuffer[8..]);
if (width is 0 or > int.MaxValue || height is 0 or > int.MaxValue)
{
throw new InvalidImageContentException("The image spatial extents property has invalid dimensions.");
}
properties.Add(new KeyValuePair<Heif4CharCode, object>(Heif4CharCode.Ispe, new Size((int)width, (int)height)));
break;
case Heif4CharCode.Pasp:
EnsureBufferRemaining(boxBuffer, 0, 8, "pixel aspect ratio");
@ -797,6 +811,18 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
properties.Add(new KeyValuePair<Heif4CharCode, object>(Heif4CharCode.Pixi, new int[] { channelCount, bitsPerPixel }));
break;
case Heif4CharCode.AuxC:
EnsureBufferRemaining(boxBuffer, 0, 5, "auxiliary type");
if (boxBuffer[0] != 0)
{
throw new InvalidImageContentException($"The auxiliary type property has unsupported version {boxBuffer[0]}.");
}
// aux_type is a required null-terminated string. Any remaining bytes are the registered
// auxiliary subtype payload, which is not needed to identify an alpha image plane.
string auxiliaryType = ReadNullTerminatedString(boxBuffer[4..], out _);
properties.Add(new KeyValuePair<Heif4CharCode, object>(Heif4CharCode.AuxC, auxiliaryType));
break;
case Heif4CharCode.Colr:
EnsureBufferRemaining(boxBuffer, 0, 4, "color information");
@ -926,6 +952,14 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
item.ChannelCount = values[0];
item.BitsPerPixel = values[1];
break;
case Heif4CharCode.AuxC:
if (item.AuxiliaryType is not null)
{
throw new InvalidImageContentException($"Item {itemId} associates more than one auxiliary type property.");
}
item.AuxiliaryType = (string)prop.Value;
break;
}
}
}
@ -1195,10 +1229,7 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
throw new ImageFormatException("No primary HEIF item defined.");
}
IHeifItemDecoder<TPixel>? itemDecoder = rootItem.Type == Heif4CharCode.Grid
&& this.FindDecodableGridTile<TPixel>(rootItem) is not null
? new GridHeifItemDecoder<TPixel>(this.configuration, this.items, this.itemLinks, buffers)
: HeifCompressionFactory.GetDecoder<TPixel>(rootItem.Type);
IHeifItemDecoder<TPixel>? itemDecoder = this.GetItemDecoder<TPixel>(rootItem, buffers);
HeifItem itemToDecode = rootItem;
if (itemDecoder is null)
@ -1217,18 +1248,178 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
throw new ImageFormatException("No decodable item found inside this HEIF container.");
}
if (!buffers.TryGetValue(itemToDecode.Id, out IMemoryOwner<byte>? itemMemory))
Image<TPixel> image = this.DecodeImageItem(itemToDecode, itemDecoder, buffers);
try
{
throw new InvalidImageContentException($"Item {itemToDecode.Id} has no data extents.");
using Image<L16>? alphaImage = this.DecodeAlphaPlane(itemToDecode, buffers, out bool alphaPremultiplied);
if (alphaImage is not null)
{
this.ApplyAlpha(image, alphaImage, alphaPremultiplied);
}
// The decoder determines the compression of the pixels that were actually returned, including grid tiles
// and a thumbnail fallback when the primary image compression is not available.
HeifMetadata meta = image.Metadata.GetHeifMetadata();
meta.CompressionMethod = itemDecoder.CompressionMethod;
meta.HasAlpha = alphaImage is not null;
return image;
}
catch
{
// Ownership transfers to the caller only after every auxiliary plane has been composed successfully.
image.Dispose();
throw;
}
}
Image<TPixel> image = itemDecoder.DecodeItemData(this.configuration, itemToDecode, itemMemory.GetSpan());
/// <summary>
/// Selects the registered coded-image or grid decoder for an image item.
/// </summary>
/// <typeparam name="TPixel">The destination pixel format.</typeparam>
/// <param name="item">The coded or derived image item.</param>
/// <param name="buffers">The assembled payloads available to a grid decoder and its tiles.</param>
/// <returns>The selected decoder, or <see langword="null"/> when the item cannot be reconstructed.</returns>
private IHeifItemDecoder<TPixel>? GetItemDecoder<TPixel>(HeifItem item, DisposableDictionary<uint, IMemoryOwner<byte>> buffers)
where TPixel : unmanaged, IPixel<TPixel>
=> item.Type == Heif4CharCode.Grid && this.FindDecodableGridTile<TPixel>(item) is not null
? new GridHeifItemDecoder<TPixel>(this.configuration, this.items, this.itemLinks, buffers)
: HeifCompressionFactory.GetDecoder<TPixel>(item.Type);
// The decoder determines the compression of the pixels that were actually returned, including grid tiles
// and a thumbnail fallback when the primary image compression is not available.
HeifMetadata meta = image.Metadata.GetHeifMetadata();
meta.CompressionMethod = itemDecoder.CompressionMethod;
return image;
/// <summary>
/// Decodes one image item from its assembled payload.
/// </summary>
/// <typeparam name="TPixel">The destination pixel format.</typeparam>
/// <param name="item">The image item to decode.</param>
/// <param name="decoder">The decoder selected for the item.</param>
/// <param name="buffers">The assembled item payloads.</param>
/// <returns>The decoded image.</returns>
private Image<TPixel> DecodeImageItem<TPixel>(
HeifItem item,
IHeifItemDecoder<TPixel> decoder,
DisposableDictionary<uint, IMemoryOwner<byte>> buffers)
where TPixel : unmanaged, IPixel<TPixel>
{
if (!buffers.TryGetValue(item.Id, out IMemoryOwner<byte>? itemMemory))
{
throw new InvalidImageContentException($"Item {item.Id} has no data extents.");
}
Image<TPixel> image = decoder.DecodeItemData(this.configuration, item, itemMemory.GetSpan());
try
{
HeifItemDecoderUtilities.ScaleToItemExtent(image, item);
return image;
}
catch
{
image.Dispose();
throw;
}
}
/// <summary>
/// Decodes the direct or per-grid-tile alpha auxiliary plane associated with a color image item.
/// </summary>
/// <param name="colorItem">The color image item whose alpha plane is requested.</param>
/// <param name="buffers">The assembled item payloads.</param>
/// <param name="premultiplied">Indicates whether the color samples are premultiplied by the decoded alpha.</param>
/// <returns>The normalized 16-bit alpha plane, or <see langword="null"/> when the item has no alpha auxiliary.</returns>
private Image<L16>? DecodeAlphaPlane(
HeifItem colorItem,
DisposableDictionary<uint, IMemoryOwner<byte>> buffers,
out bool premultiplied)
{
premultiplied = false;
HeifItem? alphaItem = this.FindAlphaItem(colorItem);
if (alphaItem is not null)
{
IHeifItemDecoder<L16>? decoder = this.GetItemDecoder<L16>(alphaItem, buffers);
if (decoder is null)
{
throw new ImageFormatException($"The alpha auxiliary item uses unsupported item type '{alphaItem.Type}'.");
}
premultiplied = this.itemLinks.Any(
link => link.Type == Heif4CharCode.Prem
&& link.SourceId == colorItem.Id
&& link.DestinationIds.Contains(alphaItem.Id));
return this.DecodeImageItem(alphaItem, decoder, buffers);
}
if (colorItem.Type != Heif4CharCode.Grid)
{
return null;
}
List<uint>? alphaTileIds = this.FindGridAlphaTiles(colorItem);
if (alphaTileIds is null)
{
return null;
}
if (!buffers.TryGetValue(colorItem.Id, out IMemoryOwner<byte>? gridMemory))
{
throw new InvalidImageContentException($"Item {colorItem.Id} has no data extents.");
}
// The color grid descriptor defines the same row/column layout and output canvas for per-tile alpha
// auxiliaries. Supplying their IDs lets the existing grid compositor preserve that normative ordering.
GridHeifItemDecoder<L16> gridDecoder = new(
this.configuration,
this.items,
this.itemLinks,
buffers,
alphaTileIds);
return gridDecoder.DecodeItemData(this.configuration, colorItem, gridMemory.GetSpan());
}
/// <summary>
/// Composes a normalized alpha plane into a decoded color image.
/// </summary>
/// <typeparam name="TPixel">The decoded color pixel format.</typeparam>
/// <param name="image">The decoded color image.</param>
/// <param name="alphaImage">The normalized 16-bit alpha plane.</param>
/// <param name="premultiplied">Whether the stored color values must be converted to unassociated alpha.</param>
private void ApplyAlpha<TPixel>(Image<TPixel> image, Image<L16> alphaImage, bool premultiplied)
where TPixel : unmanaged, IPixel<TPixel>
{
if (alphaImage.Width != image.Width || alphaImage.Height != image.Height)
{
// HEIF permits auxiliary alpha dimensions to differ from the master image. libavif uses a box filter
// for this plane scaling, which maps directly to ImageSharp's existing resampler.
alphaImage.Mutate(context => context.Resize(image.Width, image.Height, KnownResamplers.Box));
}
using IMemoryOwner<Rgba64> rowOwner = this.configuration.MemoryAllocator.Allocate<Rgba64>(image.Width);
Span<Rgba64> rgbaRow = rowOwner.GetSpan()[..image.Width];
PixelOperations<TPixel> pixelOperations = PixelOperations<TPixel>.Instance;
ImageFrame<TPixel> colorFrame = image.Frames.RootFrame;
ImageFrame<L16> alphaFrame = alphaImage.Frames.RootFrame;
for (int y = 0; y < image.Height; y++)
{
Span<TPixel> colorRow = colorFrame.PixelBuffer.DangerousGetRowSpan(y);
Span<L16> alphaRow = alphaFrame.PixelBuffer.DangerousGetRowSpan(y);
pixelOperations.ToRgba64(this.configuration, colorRow, rgbaRow);
for (int x = 0; x < image.Width; x++)
{
Rgba64 pixel = rgbaRow[x];
pixel.A = alphaRow[x].PackedValue;
if (premultiplied)
{
// libavif defines transparent premultiplied samples as transparent black. For nonzero alpha,
// reuse the packed pixel's associated-input conversion so clamping and rounding follow ImageSharp.
pixel = pixel.A == 0
? new Rgba64(0, 0, 0, 0)
: Rgba64.FromAssociatedScaledVector4(pixel.ToScaledVector4());
}
rgbaRow[x] = pixel;
}
pixelOperations.FromRgba64(this.configuration, rgbaRow, colorRow);
}
}
/// <summary>
@ -1307,6 +1498,87 @@ internal sealed class HeifDecoderCore : ImageDecoderCore
private HeifItem? FindItemById(uint itemId)
=> this.items.FirstOrDefault(item => item.Id == itemId);
/// <summary>
/// Finds the alpha auxiliary image linked to a color image item.
/// </summary>
/// <param name="colorItem">The color image item.</param>
/// <returns>The alpha auxiliary item, or <see langword="null"/> when no registered alpha relationship exists.</returns>
private HeifItem? FindAlphaItem(HeifItem colorItem)
{
HeifItem? alphaItem = null;
foreach (HeifItemLink link in this.itemLinks)
{
if (link.Type != Heif4CharCode.Auxl || !link.DestinationIds.Contains(colorItem.Id))
{
continue;
}
HeifItem candidate = this.FindItemById(link.SourceId)!;
if (!HeifConstants.IsAlphaAuxiliaryType(candidate.AuxiliaryType))
{
continue;
}
if (alphaItem is not null && alphaItem.Id != candidate.Id)
{
throw new InvalidImageContentException($"Item {colorItem.Id} has more than one alpha auxiliary image.");
}
alphaItem = candidate;
}
return alphaItem;
}
/// <summary>
/// Resolves one alpha auxiliary image for each tile of a color grid.
/// </summary>
/// <param name="gridItem">The color grid whose tile order defines the alpha grid.</param>
/// <returns>
/// The row-major alpha tile identifiers, or <see langword="null"/> when any color tile has no alpha auxiliary.
/// </returns>
private List<uint>? FindGridAlphaTiles(HeifItem gridItem)
{
List<uint> colorTileIds = [];
foreach (HeifItemLink link in this.itemLinks)
{
if (link.Type == Heif4CharCode.Dimg && link.SourceId == gridItem.Id)
{
colorTileIds.AddRange(link.DestinationIds);
}
}
if (colorTileIds.Count == 0)
{
return null;
}
List<uint> alphaTileIds = new(colorTileIds.Count);
foreach (uint colorTileId in colorTileIds)
{
HeifItem colorTile = this.FindItemById(colorTileId)!;
HeifItem? alphaTile = this.FindAlphaItem(colorTile);
if (alphaTile is null)
{
// A partial set cannot describe an alpha plane for the complete grid. libavif treats this case as
// an opaque image rather than mixing opaque cells with auxiliary alpha cells.
return null;
}
bool alphaIsDerivedTile = this.itemLinks.Any(
link => link.Type == Heif4CharCode.Dimg && link.DestinationIds.Contains(alphaTile.Id));
if (alphaIsDerivedTile)
{
throw new InvalidImageContentException($"Alpha auxiliary item {alphaTile.Id} is already a derived-image tile.");
}
alphaTileIds.Add(alphaTile.Id);
}
return alphaTileIds;
}
/// <summary>
/// Finds the first tile of a grid when every referenced tile uses a registered still-image decoder.
/// </summary>

5
src/ImageSharp/Formats/Heif/HeifItem.cs

@ -45,6 +45,11 @@ internal class HeifItem(Heif4CharCode type, uint id)
/// </summary>
public string? UriType { get; set; }
/// <summary>
/// Gets or sets the registered auxiliary type associated with this image item.
/// </summary>
public string? AuxiliaryType { get; set; }
/// <summary>
/// Gets or sets the aspect ratio of the pixels.
/// </summary>

33
src/ImageSharp/Formats/Heif/HeifItemDecoderUtilities.cs

@ -0,0 +1,33 @@
// Copyright (c) Six Labors.
// Licensed under the Six Labors Split License.
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
namespace SixLabors.ImageSharp.Formats.Heif;
/// <summary>
/// Provides shared presentation operations for decoded HEIF image items.
/// </summary>
internal static class HeifItemDecoderUtilities
{
/// <summary>
/// Scales a decoded image to the spatial extent associated with its image item.
/// </summary>
/// <typeparam name="TPixel">The decoded pixel format.</typeparam>
/// <param name="image">The decoded image.</param>
/// <param name="item">The image item that defines the presented spatial extent.</param>
public static void ScaleToItemExtent<TPixel>(Image<TPixel> image, HeifItem item)
where TPixel : unmanaged, IPixel<TPixel>
{
Size extent = item.Extent;
if (extent == default || (image.Width == extent.Width && image.Height == extent.Height))
{
return;
}
// libavif applies box filtering when coded dimensions differ from an item's ispe dimensions. Reusing the
// same ImageSharp resampler keeps direct images and grid tiles on one presentation path.
image.Mutate(context => context.Resize(extent.Width, extent.Height, KnownResamplers.Box));
}
}

23
src/ImageSharp/Formats/Heif/HeifMetadata.cs

@ -23,29 +23,40 @@ public class HeifMetadata : IFormatMetadata<HeifMetadata>
/// </summary>
/// <param name="other">The metadata to create an instance from.</param>
private HeifMetadata(HeifMetadata other)
=> this.CompressionMethod = other.CompressionMethod;
{
this.CompressionMethod = other.CompressionMethod;
this.HasAlpha = other.HasAlpha;
}
/// <summary>
/// Gets or sets the compression method used for the primary frame.
/// </summary>
public HeifCompressionMethod CompressionMethod { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the primary image has an alpha channel.
/// </summary>
public bool HasAlpha { get; set; }
/// <inheritdoc/>
public static HeifMetadata FromFormatConnectingMetadata(FormatConnectingMetadata metadata) => new()
{
CompressionMethod = HeifCompressionMethod.LegacyJpeg
CompressionMethod = HeifCompressionMethod.LegacyJpeg,
HasAlpha = metadata.PixelTypeInfo.AlphaRepresentation != PixelAlphaRepresentation.None
};
/// <inheritdoc/>
public PixelTypeInfo GetPixelTypeInfo()
{
int bpp = 8;
PixelColorType colorType = PixelColorType.RGB;
PixelComponentInfo info = PixelComponentInfo.Create(3, bpp, 8, 8, 8);
int bpp = this.HasAlpha ? 32 : 24;
PixelColorType colorType = this.HasAlpha ? PixelColorType.RGB | PixelColorType.Alpha : PixelColorType.RGB;
PixelComponentInfo info = this.HasAlpha
? PixelComponentInfo.Create(4, bpp, 8, 8, 8, 8)
: PixelComponentInfo.Create(3, bpp, 8, 8, 8);
return new PixelTypeInfo(bpp)
{
AlphaRepresentation = PixelAlphaRepresentation.None,
AlphaRepresentation = this.HasAlpha ? PixelAlphaRepresentation.Unassociated : PixelAlphaRepresentation.None,
ColorType = colorType,
ComponentInfo = info,
};

Loading…
Cancel
Save