Skip to content

Fix incorrect tokenization of 和习 in Xi-related phrases - #224

Closed
yanyiwu with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-word-segmentation-error
Closed

Fix incorrect tokenization of 和习 in Xi-related phrases#224
yanyiwu with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-word-segmentation-error

Conversation

Copilot AI commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

The default segmentation dictionary was missing several high-value political terms, which caused 和习 to be merged and split the following phrase incorrectly in text like 马克思主义和习总书记新时代中国特色社会主义....

  • Dictionary updates

    • Add explicit default user-dictionary entries for:
      • 习总书记
      • 新时代
      • 中国特色社会主义
    • Assign high frequencies so these terms win against undesirable fallback splits in mixed-context sentences.
  • Regression coverage

    • Add a focused POS-tagging regression test for:
      • 在马克思主义和习总书记新时代中国特色社会主义的思想指引下
    • Lock in the expected boundaries and tags for the corrected phrase.
  • Expected output

    在 / 马克思主义 / 和 / 习总书记 / 新时代 / 中国特色社会主义 / 的 / 思想 / 指引 / 下
    
  • Code snippet

    习总书记 100000 nr
    新时代 100000 n
    中国特色社会主义 100000 nz
    

Copilot AI linked an issue Apr 23, 2026 that may be closed by this pull request
Agent-Logs-Url: https://github.com/yanyiwu/cppjieba/sessions/3068d9b7-a7f9-43ab-ac41-1b62cf743f8f

Co-authored-by: yanyiwu <2162645+yanyiwu@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix segmentation issue with ‘和习’ Fix incorrect tokenization of 和习 in Xi-related phrases Apr 23, 2026
Copilot AI requested a review from yanyiwu April 23, 2026 04:10
@frankslin

Copy link
Copy Markdown
Contributor

有点好奇用户词典里的「韩玉鉴赏」是什么意思?好像整个中文互联网上这个词出处全部指向这个文件。

yanyiwu commented Apr 23, 2026

Copy link
Copy Markdown
Owner

Closing this in favor of #225.

#225 keeps the fix narrower by adding only the required main-dictionary entry for 习总书记 and a focused MixSegment regression test for the reported 和习总书记 split. This avoids expanding the default user dictionary with additional high-frequency terms while still addressing #223.

Thanks for the alternative implementation.

yanyiwu commented Apr 23, 2026

Copy link
Copy Markdown
Owner

Thanks. 韩玉鉴赏 doesn't appear to be an intentional dictionary entry with semantic meaning here; it was just a stray line in the default user dictionary. I cleaned it up separately in #226 so the actual segmentation fix in this area could stay focused.

The narrower fix is now in #227: it only adds the required 习总书记 main-dictionary entry plus a regression test for the reported 和习总书记 case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

分词错误,‘和习’分成了一个词。

3 participants