Fix TextChunker orphan chunk gluing using word count instead of token count

In ProcessParagraphs, the orphan chunk merging logic was splitting
strings by spaces to get word counts and comparing those against the
token limit. This caused merged chunks to exceed the target token count
when the token counter produces different counts than naive word splitting.

Use GetTokenCount() consistently, matching how the rest of the method
already measures chunk sizes.

Fixes #13713

Author: joaquinhuigomez

dotnet/src/SemanticKernel.Core/Text/TextChunker.cs

@@ -192,18 +192,12 @@ private static List<string> ProcessParagraphs(List<string> paragraphs, int adjus
 
             if (GetTokenCount(lastParagraph, tokenCounter) < adjustedMaxTokensPerParagraph / 4)
             {
-                var lastParagraphTokens = lastParagraph.Split(s_spaceChar, StringSplitOptions.RemoveEmptyEntries);
-                var secondLastParagraphTokens = secondLastParagraph.Split(s_spaceChar, StringSplitOptions.RemoveEmptyEntries);
+                var lastParagraphTokenCount = GetTokenCount(lastParagraph, tokenCounter);
+                var secondLastParagraphTokenCount = GetTokenCount(secondLastParagraph, tokenCounter);
 
-                var lastParagraphTokensCount = lastParagraphTokens.Length;
-                var secondLastParagraphTokensCount = secondLastParagraphTokens.Length;
-
-                if (lastParagraphTokensCount + secondLastParagraphTokensCount <= adjustedMaxTokensPerParagraph)
+                if (lastParagraphTokenCount + secondLastParagraphTokenCount <= adjustedMaxTokensPerParagraph)
                 {
-                    var newSecondLastParagraph = string.Join(" ", secondLastParagraphTokens);
-                    var newLastParagraph = string.Join(" ", lastParagraphTokens);
-
-                    paragraphs[paragraphs.Count - 2] = $"{newSecondLastParagraph} {newLastParagraph}";
+                    paragraphs[paragraphs.Count - 2] = $"{secondLastParagraph} {lastParagraph}";
                     paragraphs.RemoveAt(paragraphs.Count - 1);
                 }
             }