Avalonia

Commit Graph

Author	SHA1	Message	Date
Benedikt Stebner	acc1494308	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875 ) * Pack the segmentation properties into one Unicode trie Grapheme, indic-conjunct, word, line and sentence break classes now share a single SegmentationTrie, together with the Emoji, Emoji_Presentation and Default_Ignorable_Code_Point flags that rode in the grapheme word. The UnicodeDataTrie word keeps general category, script and script extensions. - SegmentationTrieGenerator replaces GraphemeBreakClassTrieGenerator and emits Segmentation.trie.cs; UnicodeData.trie.cs is regenerated without the break fields it no longer carries. - SentenceBreakClass data and Codepoint.SentenceBreakClass expose UAX-29 sentence classes to callers. - WordSegment carries the source text slice alongside the code-unit and code-point readouts, matching the shape newer segment types use; the existing constructor and members stay for compatibility. Data regenerated against UCD 17.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add UAX-29 sentence segmentation SentenceBreakEnumerator walks sentence boundaries over a UTF-16 span (rules SB1-SB11 and SB998), yielding SentenceSegment values of offset plus text slice. Sentence is the one segmentation granularity accessibility clients and IME navigation ask for that had no implementation, so callers had to approximate it with punctuation heuristics. The conformance suite runs against the Unicode SentenceBreakTest.txt data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make sentence segmentation linear in the text length Every rule that needed left-hand context rescanned the text backwards from the current position, so a run of spaces or closing brackets was walked once per character in it. Enumerating 40k spaces took 5.5 s; 40k closing brackets took 8.1 s, and both grew quadratically. - Fold the left-hand context into the enumerator as it advances: the last two significant classes for SB6/SB7, and a stage counter over the (STerm \| ATerm) Close* Sp* (Sep \| CR \| LF)? grammar for SB8-SB11. - Memoize the SB8 lookahead. A scan answers every start position it passed over, so the Close* Sp* run after an ATerm is scanned once, not once per position. - Classify with bitmasks, matching LineBreakEnumerator. Both pathological inputs now finish in 0.1 ms at 40k characters. Output is unchanged: the UCD SentenceBreakTest.txt suite passes, and 1.4M random strings over the break classes segment identically to the previous implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Read the word segment length from the property valid for every segment WordSegment.Text is empty for segments built through the code-unit constructor that shipped in 12.1, so Text.Length holds only for segments that come from WordBreakEnumerator. Length is set by both constructors. - Measure the ellipsis break position from Length again. - Say on the Text property itself, where it is read, that it can be empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Benchmark word and sentence segmentation across text shapes The break enumerator benchmark covered three enumerators over random codepoint distributions, which measures trie lookup cost but not the per-character cost of the rules themselves. Long runs of one class are where a segmentation algorithm degrades, and nothing measured them. - Add the sentence enumerator to UnicodeBreakEnumeratorBenchmark. - Add UnicodeSegmentationScalingBenchmark: prose, Han, ZWJ emoji, and runs of spaces, closing punctuation, terminator-plus-spaces and regional indicators, each at two lengths so the cost per character is comparable between them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document the sentence break classes and trim the break unit SentenceBreakClass is public API with nothing but the UCD aliases to go on, so a caller reading Codepoint.SentenceBreakClass has to consult UAX #29 to learn what a value means or which rule reacts to it. - Describe the type and every member in terms of what the sentence rules do with it, keeping the UCD alias as a trailing comment. - Give SentenceBreakUnit a primary constructor and drop its Codepoint property, which nothing reads now that the class is stored directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Clean up documentation for SentenceBreakClass Removed remarks from the SentenceBreakClass documentation. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Benedikt Stebner	42bba0b859	[Text] Unicode and text layout optimizations (#21400 ) * perf(text): O(1) width queries on ShapedBuffer, cached LB-class lookup - ShapedBuffer: lazy, pooled cluster-prefix cache shared across Split; adds TotalGlyphAdvance / MeasureCharactersThatFit. - GlyphRun/TextFormatterImpl/Skia GlyphRunImpl: read the cache, drop duplicate scans; SplitTextRuns now returns firstLength via out. - LineBreakEnumerator: cache Next/PreviousClass per advance; remove unused LineBreakPairTable. - Tests + benchmarks for the new paths and a dotnet-trace harness. ~1.32x faster / -5% alloc on the emoji-wrap micro-benchmark; no observable behaviour or public API change. * perf(TextFormatting): inline LineBreakEnumerator rule dispatch Replace the BreakUnitDelegate[] s_rules array dispatch in LineBreakEnumerator.ExecuteRules with a sequence of direct static calls and a `goto Done` early-exit. This removes 42 indirect calls per codepoint and lets the JIT reason across rule boundaries, which in turn makes [MethodImpl(AggressiveInlining)] meaningful — JIT cannot inline through delegate.Invoke, so the attribute was a no-op in the previous shape. Selectively apply AggressiveInlining to the small single-condition rules (LB03, LB04, LB06, LB07, LB08a, LB11–LB15d, LB18, LB20, LB21b, LB22, LB29, LB31) and let the JIT decide on the larger ones to avoid bloating the merged ExecuteRules. The static BreakUnitDelegate[] s_rules array is removed. Benchmarks (BDN default job, --inProcess, N=13–22, rel. StdDev <1.5%): UnicodeBreakEnumeratorBenchmark.LineBreakEnumerator_Sequence Ascii 154.6 µs -> 24.84 µs (6.22x) Bmp 190.2 µs -> 32.81 µs (5.80x) Supplementary 227.6 µs -> 37.56 µs (6.06x) TextLayoutProfile.BuildEmojisWrapped Before (branch, pre-inline) 804.5 µs / 570.15 KB After (branch, post-inline) 547.0 µs / 570.15 KB (-32.0%) vs upstream/master (1061 µs) -> ~1.94x total speedup No allocation change; CPU-only dispatch reshape. All LineBreakEnumerator unit tests pass (5/5). * Refactor ShapedBuffer to share pool storage via IRef + generation counter Wrap the ArrayPool-rented glyph and cluster-cache arrays in a small PooledArray<T> disposable and expose them through IRef<T>. Split children and WithBidiLevel aliases now Clone() the refs instead of borrowing raw pool arrays, so the backing storage survives until every sibling has been disposed - eliminating the UAF risk that existed when a parent was disposed before its children. Add a per-glyph-holder generation counter (Volatile.Read / Interlocked .Increment). The indexer setter bumps the counter on every write, and EnsureClusterCache compares its recorded generation against the holder's current value, rebuilding on mismatch. This lets us drop the previous "no mutation after Split/WithBidiLevel" contract: mutations performed through any sibling now propagate to the others' caches transparently. Dispose is made idempotent via a _disposed guard so overlapping cache eviction and TextLine teardown only release the IRefs once. Adds ShapedBufferSharedStorageTests covering sibling lifetime, Dispose idempotency, and generation-driven cache invalidation across Split children and WithBidiLevel aliases. * test(ShapedBuffer): cover cluster-cache sharing across Split/WithBidiLevel aliases Add a regression test hook (`ClusterPrefix`) exposing the backing cluster-prefix array reference, plus two tests that mutate a parent buffer before aliasing it and assert the alias reuses the parent's pooled prefix array instead of rebuilding. Guards against forgetting to propagate `_cacheGeneration` to alias buffers, which would silently defeat the cached-split fast path. * Correctly use MathUtilities.LessThanOrClose --------- Co-authored-by: Julien Lebosquain <julien@lebosquain.net>	4 months ago

Author

SHA1

Message

Date

Benedikt Stebner

acc1494308

Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875 )

* Pack the segmentation properties into one Unicode trie

Grapheme, indic-conjunct, word, line and sentence break classes now share a
single SegmentationTrie, together with the Emoji, Emoji_Presentation and
Default_Ignorable_Code_Point flags that rode in the grapheme word. The
UnicodeDataTrie word keeps general category, script and script extensions.

- SegmentationTrieGenerator replaces GraphemeBreakClassTrieGenerator and emits
  Segmentation.trie.cs; UnicodeData.trie.cs is regenerated without the break
  fields it no longer carries.
- SentenceBreakClass data and Codepoint.SentenceBreakClass expose UAX-29
  sentence classes to callers.
- WordSegment carries the source text slice alongside the code-unit and
  code-point readouts, matching the shape newer segment types use; the existing
  constructor and members stay for compatibility.

Data regenerated against UCD 17.0.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Add UAX-29 sentence segmentation

SentenceBreakEnumerator walks sentence boundaries over a UTF-16 span (rules
SB1-SB11 and SB998), yielding SentenceSegment values of offset plus text slice.
Sentence is the one segmentation granularity accessibility clients and IME
navigation ask for that had no implementation, so callers had to approximate it
with punctuation heuristics.

The conformance suite runs against the Unicode SentenceBreakTest.txt data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Make sentence segmentation linear in the text length

Every rule that needed left-hand context rescanned the text backwards from
the current position, so a run of spaces or closing brackets was walked
once per character in it. Enumerating 40k spaces took 5.5 s; 40k closing
brackets took 8.1 s, and both grew quadratically.

- Fold the left-hand context into the enumerator as it advances: the last
  two significant classes for SB6/SB7, and a stage counter over the
  (STerm | ATerm) Close* Sp* (Sep | CR | LF)? grammar for SB8-SB11.
- Memoize the SB8 lookahead. A scan answers every start position it passed
  over, so the Close* Sp* run after an ATerm is scanned once, not once per
  position.
- Classify with bitmasks, matching LineBreakEnumerator.

Both pathological inputs now finish in 0.1 ms at 40k characters. Output is
unchanged: the UCD SentenceBreakTest.txt suite passes, and 1.4M random
strings over the break classes segment identically to the previous
implementation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Read the word segment length from the property valid for every segment

WordSegment.Text is empty for segments built through the code-unit
constructor that shipped in 12.1, so Text.Length holds only for segments
that come from WordBreakEnumerator. Length is set by both constructors.

- Measure the ellipsis break position from Length again.
- Say on the Text property itself, where it is read, that it can be empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Benchmark word and sentence segmentation across text shapes

The break enumerator benchmark covered three enumerators over random
codepoint distributions, which measures trie lookup cost but not the
per-character cost of the rules themselves. Long runs of one class are
where a segmentation algorithm degrades, and nothing measured them.

- Add the sentence enumerator to UnicodeBreakEnumeratorBenchmark.
- Add UnicodeSegmentationScalingBenchmark: prose, Han, ZWJ emoji, and runs
  of spaces, closing punctuation, terminator-plus-spaces and regional
  indicators, each at two lengths so the cost per character is comparable
  between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Document the sentence break classes and trim the break unit

SentenceBreakClass is public API with nothing but the UCD aliases to go on,
so a caller reading Codepoint.SentenceBreakClass has to consult UAX #29 to
learn what a value means or which rule reacts to it.

- Describe the type and every member in terms of what the sentence rules do
  with it, keeping the UCD alias as a trailing comment.
- Give SentenceBreakUnit a primary constructor and drop its Codepoint
  property, which nothing reads now that the class is stored directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Clean up documentation for SentenceBreakClass

Removed remarks from the SentenceBreakClass documentation.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

1 week ago

Benedikt Stebner

42bba0b859

[Text] Unicode and text layout optimizations (#21400 )

* perf(text): O(1) width queries on ShapedBuffer, cached LB-class lookup

- ShapedBuffer: lazy, pooled cluster-prefix cache shared across Split;
  adds TotalGlyphAdvance / MeasureCharactersThatFit.
- GlyphRun/TextFormatterImpl/Skia GlyphRunImpl: read the cache, drop
  duplicate scans; SplitTextRuns now returns firstLength via out.
- LineBreakEnumerator: cache Next/PreviousClass per advance; remove
  unused LineBreakPairTable.
- Tests + benchmarks for the new paths and a dotnet-trace harness.

~1.32x faster / -5% alloc on the emoji-wrap micro-benchmark; no
observable behaviour or public API change.

* perf(TextFormatting): inline LineBreakEnumerator rule dispatch

Replace the BreakUnitDelegate[] s_rules array dispatch in
LineBreakEnumerator.ExecuteRules with a sequence of direct static
calls and a `goto Done` early-exit. This removes 42 indirect calls
per codepoint and lets the JIT reason across rule boundaries, which
in turn makes [MethodImpl(AggressiveInlining)] meaningful — JIT
cannot inline through delegate.Invoke, so the attribute was a no-op
in the previous shape.

Selectively apply AggressiveInlining to the small single-condition
rules (LB03, LB04, LB06, LB07, LB08a, LB11–LB15d, LB18, LB20,
LB21b, LB22, LB29, LB31) and let the JIT decide on the larger ones
to avoid bloating the merged ExecuteRules.

The static BreakUnitDelegate[] s_rules array is removed.

Benchmarks (BDN default job, --inProcess, N=13–22, rel. StdDev <1.5%):

  UnicodeBreakEnumeratorBenchmark.LineBreakEnumerator_Sequence
    Ascii          154.6 µs -> 24.84 µs  (6.22x)
    Bmp            190.2 µs -> 32.81 µs  (5.80x)
    Supplementary  227.6 µs -> 37.56 µs  (6.06x)

  TextLayoutProfile.BuildEmojisWrapped
    Before (branch, pre-inline)  804.5 µs / 570.15 KB
    After  (branch, post-inline) 547.0 µs / 570.15 KB  (-32.0%)
    vs upstream/master (1061 µs) -> ~1.94x total speedup

No allocation change; CPU-only dispatch reshape. All
LineBreakEnumerator unit tests pass (5/5).

* Refactor ShapedBuffer to share pool storage via IRef + generation counter

Wrap the ArrayPool-rented glyph and cluster-cache arrays in a small
PooledArray<T> disposable and expose them through IRef<T>. Split children
and WithBidiLevel aliases now Clone() the refs instead of borrowing raw
pool arrays, so the backing storage survives until every sibling has
been disposed - eliminating the UAF risk that existed when a parent was
disposed before its children.

Add a per-glyph-holder generation counter (Volatile.Read / Interlocked
.Increment). The indexer setter bumps the counter on every write, and
EnsureClusterCache compares its recorded generation against the holder's
current value, rebuilding on mismatch. This lets us drop the previous
"no mutation after Split/WithBidiLevel" contract: mutations performed
through any sibling now propagate to the others' caches transparently.

Dispose is made idempotent via a _disposed guard so overlapping cache
eviction and TextLine teardown only release the IRefs once.

Adds ShapedBufferSharedStorageTests covering sibling lifetime,
Dispose idempotency, and generation-driven cache invalidation across
Split children and WithBidiLevel aliases.

* test(ShapedBuffer): cover cluster-cache sharing across Split/WithBidiLevel aliases

Add a regression test hook (`ClusterPrefix`) exposing the
backing cluster-prefix array reference, plus two tests that mutate a
parent buffer before aliasing it and assert the alias reuses the
parent's pooled prefix array instead of rebuilding. Guards against
forgetting to propagate `_cacheGeneration` to alias buffers, which
would silently defeat the cached-split fast path.

* Correctly use MathUtilities.LessThanOrClose

---------

Co-authored-by: Julien Lebosquain <julien@lebosquain.net>

4 months ago

2 Commits (main)