Skip to content

Commit d0287f2

Browse files
committed
fix(tokenize): correct capture group reference in website regex
The websites regex /[.](com|net|org|io|gov|edu|me)/g has only one capture group for the TLD, but was incorrectly referencing $2. This caused URLs to be corrupted with literal '$2' text. Changed $2 to $1 to correctly reference the TLD capture group.
1 parent 0bb2745 commit d0287f2

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

agents/src/tokenize/basic/sentence.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ export const splitSentences = (
2727
}
2828

2929
text = text.replaceAll(prefixes, '$1<prd>');
30-
text = text.replaceAll(websites, '<prd>$2');
30+
text = text.replaceAll(websites, '<prd>$1');
3131
text = text.replaceAll(new RegExp(`${digits.source}[.]${digits.source}`, 'g'), '$1<prd>$2');
3232
text = text.replaceAll(dots, (match) => '<prd>'.repeat(match.length));
3333
text = text.replaceAll('Ph.D.', 'Ph<prd>D<prd>');

0 commit comments

Comments
 (0)