Avalonia

Branch: main

Commit Graph

Author	SHA1	Message	Date
Benedikt Stebner	acc1494308	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875 ) * Pack the segmentation properties into one Unicode trie Grapheme, indic-conjunct, word, line and sentence break classes now share a single SegmentationTrie, together with the Emoji, Emoji_Presentation and Default_Ignorable_Code_Point flags that rode in the grapheme word. The UnicodeDataTrie word keeps general category, script and script extensions. - SegmentationTrieGenerator replaces GraphemeBreakClassTrieGenerator and emits Segmentation.trie.cs; UnicodeData.trie.cs is regenerated without the break fields it no longer carries. - SentenceBreakClass data and Codepoint.SentenceBreakClass expose UAX-29 sentence classes to callers. - WordSegment carries the source text slice alongside the code-unit and code-point readouts, matching the shape newer segment types use; the existing constructor and members stay for compatibility. Data regenerated against UCD 17.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add UAX-29 sentence segmentation SentenceBreakEnumerator walks sentence boundaries over a UTF-16 span (rules SB1-SB11 and SB998), yielding SentenceSegment values of offset plus text slice. Sentence is the one segmentation granularity accessibility clients and IME navigation ask for that had no implementation, so callers had to approximate it with punctuation heuristics. The conformance suite runs against the Unicode SentenceBreakTest.txt data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make sentence segmentation linear in the text length Every rule that needed left-hand context rescanned the text backwards from the current position, so a run of spaces or closing brackets was walked once per character in it. Enumerating 40k spaces took 5.5 s; 40k closing brackets took 8.1 s, and both grew quadratically. - Fold the left-hand context into the enumerator as it advances: the last two significant classes for SB6/SB7, and a stage counter over the (STerm \| ATerm) Close* Sp* (Sep \| CR \| LF)? grammar for SB8-SB11. - Memoize the SB8 lookahead. A scan answers every start position it passed over, so the Close* Sp* run after an ATerm is scanned once, not once per position. - Classify with bitmasks, matching LineBreakEnumerator. Both pathological inputs now finish in 0.1 ms at 40k characters. Output is unchanged: the UCD SentenceBreakTest.txt suite passes, and 1.4M random strings over the break classes segment identically to the previous implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Read the word segment length from the property valid for every segment WordSegment.Text is empty for segments built through the code-unit constructor that shipped in 12.1, so Text.Length holds only for segments that come from WordBreakEnumerator. Length is set by both constructors. - Measure the ellipsis break position from Length again. - Say on the Text property itself, where it is read, that it can be empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Benchmark word and sentence segmentation across text shapes The break enumerator benchmark covered three enumerators over random codepoint distributions, which measures trie lookup cost but not the per-character cost of the rules themselves. Long runs of one class are where a segmentation algorithm degrades, and nothing measured them. - Add the sentence enumerator to UnicodeBreakEnumeratorBenchmark. - Add UnicodeSegmentationScalingBenchmark: prose, Han, ZWJ emoji, and runs of spaces, closing punctuation, terminator-plus-spaces and regional indicators, each at two lengths so the cost per character is comparable between them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document the sentence break classes and trim the break unit SentenceBreakClass is public API with nothing but the UCD aliases to go on, so a caller reading Codepoint.SentenceBreakClass has to consult UAX #29 to learn what a value means or which rule reacts to it. - Describe the type and every member in terms of what the sentence rules do with it, keeping the UCD alias as a trailing comment. - Give SentenceBreakUnit a primary constructor and drop its Codepoint property, which nothing reads now that the class is stored directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Clean up documentation for SentenceBreakClass Removed remarks from the SentenceBreakClass documentation. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	1 week ago

Author

SHA1

Message

Date

Benedikt Stebner

acc1494308

Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875 )

* Pack the segmentation properties into one Unicode trie

Grapheme, indic-conjunct, word, line and sentence break classes now share a
single SegmentationTrie, together with the Emoji, Emoji_Presentation and
Default_Ignorable_Code_Point flags that rode in the grapheme word. The
UnicodeDataTrie word keeps general category, script and script extensions.

- SegmentationTrieGenerator replaces GraphemeBreakClassTrieGenerator and emits
  Segmentation.trie.cs; UnicodeData.trie.cs is regenerated without the break
  fields it no longer carries.
- SentenceBreakClass data and Codepoint.SentenceBreakClass expose UAX-29
  sentence classes to callers.
- WordSegment carries the source text slice alongside the code-unit and
  code-point readouts, matching the shape newer segment types use; the existing
  constructor and members stay for compatibility.

Data regenerated against UCD 17.0.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Add UAX-29 sentence segmentation

SentenceBreakEnumerator walks sentence boundaries over a UTF-16 span (rules
SB1-SB11 and SB998), yielding SentenceSegment values of offset plus text slice.
Sentence is the one segmentation granularity accessibility clients and IME
navigation ask for that had no implementation, so callers had to approximate it
with punctuation heuristics.

The conformance suite runs against the Unicode SentenceBreakTest.txt data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Make sentence segmentation linear in the text length

Every rule that needed left-hand context rescanned the text backwards from
the current position, so a run of spaces or closing brackets was walked
once per character in it. Enumerating 40k spaces took 5.5 s; 40k closing
brackets took 8.1 s, and both grew quadratically.

- Fold the left-hand context into the enumerator as it advances: the last
  two significant classes for SB6/SB7, and a stage counter over the
  (STerm | ATerm) Close* Sp* (Sep | CR | LF)? grammar for SB8-SB11.
- Memoize the SB8 lookahead. A scan answers every start position it passed
  over, so the Close* Sp* run after an ATerm is scanned once, not once per
  position.
- Classify with bitmasks, matching LineBreakEnumerator.

Both pathological inputs now finish in 0.1 ms at 40k characters. Output is
unchanged: the UCD SentenceBreakTest.txt suite passes, and 1.4M random
strings over the break classes segment identically to the previous
implementation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Read the word segment length from the property valid for every segment

WordSegment.Text is empty for segments built through the code-unit
constructor that shipped in 12.1, so Text.Length holds only for segments
that come from WordBreakEnumerator. Length is set by both constructors.

- Measure the ellipsis break position from Length again.
- Say on the Text property itself, where it is read, that it can be empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Benchmark word and sentence segmentation across text shapes

The break enumerator benchmark covered three enumerators over random
codepoint distributions, which measures trie lookup cost but not the
per-character cost of the rules themselves. Long runs of one class are
where a segmentation algorithm degrades, and nothing measured them.

- Add the sentence enumerator to UnicodeBreakEnumeratorBenchmark.
- Add UnicodeSegmentationScalingBenchmark: prose, Han, ZWJ emoji, and runs
  of spaces, closing punctuation, terminator-plus-spaces and regional
  indicators, each at two lengths so the cost per character is comparable
  between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Document the sentence break classes and trim the break unit

SentenceBreakClass is public API with nothing but the UCD aliases to go on,
so a caller reading Codepoint.SentenceBreakClass has to consult UAX #29 to
learn what a value means or which rule reacts to it.

- Describe the type and every member in terms of what the sentence rules do
  with it, keeping the UCD alias as a trailing comment.
- Give SentenceBreakUnit a primary constructor and drop its Codepoint
  property, which nothing reads now that the class is stored directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Clean up documentation for SentenceBreakClass

Removed remarks from the SentenceBreakClass documentation.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

1 week ago

1 Commits (main)