Avalonia

A cross-platform UI framework for .NET

csharp c-sharp dotnet xaml avaloniaui cross-platform cross-platform-xaml avalonia gui multi-platform user-interface dotnetcore

History

Benedikt Stebner acc1494308 Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875 ) * Pack the segmentation properties into one Unicode trie Grapheme, indic-conjunct, word, line and sentence break classes now share a single SegmentationTrie, together with the Emoji, Emoji_Presentation and Default_Ignorable_Code_Point flags that rode in the grapheme word. The UnicodeDataTrie word keeps general category, script and script extensions. - SegmentationTrieGenerator replaces GraphemeBreakClassTrieGenerator and emits Segmentation.trie.cs; UnicodeData.trie.cs is regenerated without the break fields it no longer carries. - SentenceBreakClass data and Codepoint.SentenceBreakClass expose UAX-29 sentence classes to callers. - WordSegment carries the source text slice alongside the code-unit and code-point readouts, matching the shape newer segment types use; the existing constructor and members stay for compatibility. Data regenerated against UCD 17.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add UAX-29 sentence segmentation SentenceBreakEnumerator walks sentence boundaries over a UTF-16 span (rules SB1-SB11 and SB998), yielding SentenceSegment values of offset plus text slice. Sentence is the one segmentation granularity accessibility clients and IME navigation ask for that had no implementation, so callers had to approximate it with punctuation heuristics. The conformance suite runs against the Unicode SentenceBreakTest.txt data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make sentence segmentation linear in the text length Every rule that needed left-hand context rescanned the text backwards from the current position, so a run of spaces or closing brackets was walked once per character in it. Enumerating 40k spaces took 5.5 s; 40k closing brackets took 8.1 s, and both grew quadratically. - Fold the left-hand context into the enumerator as it advances: the last two significant classes for SB6/SB7, and a stage counter over the (STerm \| ATerm) Close* Sp* (Sep \| CR \| LF)? grammar for SB8-SB11. - Memoize the SB8 lookahead. A scan answers every start position it passed over, so the Close* Sp* run after an ATerm is scanned once, not once per position. - Classify with bitmasks, matching LineBreakEnumerator. Both pathological inputs now finish in 0.1 ms at 40k characters. Output is unchanged: the UCD SentenceBreakTest.txt suite passes, and 1.4M random strings over the break classes segment identically to the previous implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Read the word segment length from the property valid for every segment WordSegment.Text is empty for segments built through the code-unit constructor that shipped in 12.1, so Text.Length holds only for segments that come from WordBreakEnumerator. Length is set by both constructors. - Measure the ellipsis break position from Length again. - Say on the Text property itself, where it is read, that it can be empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Benchmark word and sentence segmentation across text shapes The break enumerator benchmark covered three enumerators over random codepoint distributions, which measures trie lookup cost but not the per-character cost of the rules themselves. Long runs of one class are where a segmentation algorithm degrades, and nothing measured them. - Add the sentence enumerator to UnicodeBreakEnumeratorBenchmark. - Add UnicodeSegmentationScalingBenchmark: prose, Han, ZWJ emoji, and runs of spaces, closing punctuation, terminator-plus-spaces and regional indicators, each at two lengths so the cost per character is comparable between them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document the sentence break classes and trim the break unit SentenceBreakClass is public API with nothing but the UCD aliases to go on, so a caller reading Codepoint.SentenceBreakClass has to consult UAX #29 to learn what a value means or which rule reacts to it. - Describe the type and every member in terms of what the sentence rules do with it, keeping the UCD alias as a trailing comment. - Give SentenceBreakUnit a primary constructor and drop its Codepoint property, which nothing reads now that the class is stored directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Clean up documentation for SentenceBreakClass Removed remarks from the SentenceBreakClass documentation. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>		1 week ago
..
Unicode	[Text] Fixes culture aware font matching (#21435)	4 months ago
BiDiAlgorithmTests.cs	[Text] Implement Unicode word segmentation and update to Unicode v17 (#21361)	4 months ago
BiDiClassTestDataGenerator.cs	[Text] Move Unicode trie generation to a dedicated Nuke-driven tool (#21391)	4 months ago
BiDiClassTests.cs	[Text] Implement Unicode word segmentation and update to Unicode v17 (#21361)	4 months ago
BiDiTestDataGenerator.cs	[Text] Move Unicode trie generation to a dedicated Nuke-driven tool (#21391)	4 months ago
CodepointTests.cs	[Text] Ignore default ignorable codepoints in font fallback (#22059)	3 weeks ago
FormattedTextSourceTests.cs	Enable nullability in Base/Markup/Markup.Xaml unit tests (#20346)	9 months ago
FormattingBufferHelperTests.cs	Fixed text layout buffer reset for Stack and Dictionary (+ tests)	4 years ago
GraphemeBreakClassTrieGeneratorTests.cs	[Text] Move Unicode trie generation to a dedicated Nuke-driven tool (#21391)	4 months ago
GraphemeBreakTestDataGenerator.cs	[Text] Move Unicode trie generation to a dedicated Nuke-driven tool (#21391)	4 months ago
HarfbuzzTextShaperTests.cs	Ensure glyph clusters are slice relative (#21089)	5 months ago
LineBreakEnumeratorTests.cs	Fix line break enumerator infinite loop (#21808)	2 months ago
PropertyValueAliasHelperTests.cs	[Text] Unicode and text layout optimizations (#21400)	4 months ago
SentenceBreakEnumeratorTests.cs	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875)	1 week ago
SentenceBreakTestDataGenerator.cs	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875)	1 week ago
ShapedBufferTests.cs	[Text] Glyph Outline Part 5/13 - Shaped buffer glyph indices (#21501)	1 week ago
UnicodeDataTests.cs	[Text] Unicode and text layout optimizations (#21400)	4 months ago
UnicodeTrieTests.cs	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875)	1 week ago
Utf16UtilsTests.cs	X11 IME preedit, preedit cursor, input context improvements (#13282)	3 years ago
WordBreakEnumeratorTests.cs	Consolidate the Unicode segmentation trie and add UAX-29 sentence segmentation (#21875)	1 week ago