Improve the usability of the analysis script

[lukhnos]

Now if you type make analyze, you will see a report like this (I'm using make ANALYZE_LIMIT=10 analyze to make the report shorter):

python3 -m curation.analyzers.find_cover_issues \
	--input data.txt \
	--limit 10

------------------------------------------------------------------------
Summary
------------------------------------------------------------------------
 26651 unigrams with one character
145111 unigrams with multiple characters
  1447 emojis
    49 macros

------------------------------------------------------------------------
Multi-Character Phrases with Issues
------------------------------------------------------------------------
22934 unigrams that are not the top candidate (15.8% of unigrams)

of which:
 22906 2-character unigrams
    28 3-character unigrams
     0 4-character unigrams
     0 5-character unigrams
     0 6-character unigrams

2209 unigrams also compete with unigrams with top-ranking characters
5571 unigrams whose scores are lower than their identical components

------------------------------------------------------------------------
Top Insufficient 2-Character Unigrams
------------------------------------------------------------------------
insufficient 它的 -4.2064 < 他+的 -4.1459 ㄊㄚ-ㄉㄜ˙
insufficient 她的 -4.3475 < 他+的 -4.1459 ㄊㄚ-ㄉㄜ˙
insufficient 人事 -4.3920 < 人+是 -4.3405 ㄖㄣˊ-ㄕˋ
insufficient 曉得 -4.5578 < 小+的 -4.4707 ㄒㄧㄠˇ-ㄉㄜ˙
insufficient 一再 -4.5760 < 一+在 -4.3191 ㄧ-ㄗㄞˋ
insufficient 看得 -4.5813 < 看+的 -4.4459 ㄎㄢˋ-ㄉㄜ˙
insufficient 試試 -4.5941 < 是+是 -4.0749 ㄕˋ-ㄕˋ
insufficient 中市 -4.6110 < 中+是 -4.5405 ㄓㄨㄥ-ㄕˋ
insufficient 來得 -4.6760 < 來+的 -4.0905 ㄌㄞˊ-ㄉㄜ˙
insufficient 各式 -4.6816 < 個+是 -4.4385 ㄍㄜˋ-ㄕˋ
...and 22896 more entries suppressed

------------------------------------------------------------------------
Top Insufficient 3-Character Unigrams
------------------------------------------------------------------------
insufficient 一再的 -6.1233 < 一+在+的 -5.9410 ㄧ-ㄗㄞˋ-ㄉㄜ˙
insufficient 多得是 -6.5212 < 多+的+是 -6.3581 ㄉㄨㄛ-ㄉㄜ˙-ㄕˋ
insufficient 人事部 -6.5212 < 人+是+不 -6.5186 ㄖㄣˊ-ㄕˋ-ㄅㄨˋ
insufficient 這隻的 -6.8223 < 這+之+的 -6.6544 ㄓㄜˋ-ㄓ-ㄉㄜ˙
insufficient 再就是 -6.8223 < 在+就+是 -6.8129 ㄗㄞˋ-ㄐㄧㄡˋ-ㄕˋ
insufficient 醫學士 -6.8223 < 一+學+是 -6.7619 ㄧ-ㄒㄩㄝˊ-ㄕˋ
insufficient 門上的 -7.1233 < 們+上+的 -6.7138 ㄇㄣˊ-ㄕㄤˋ-ㄉㄜ˙
insufficient 說不得 -7.1233 < 說+不+的 -6.5223 ㄕㄨㄛ-ㄅㄨˋ-ㄉㄜ˙
insufficient 說得來 -7.1233 < 說+的+來 -6.8127 ㄕㄨㄛ-ㄉㄜ˙-ㄌㄞˊ
insufficient 紫式部 -7.1233 < 子+是+不 -7.0076 ㄗˇ-ㄕˋ-ㄅㄨˋ
...and 18 more entries suppressed

------------------------------------------------------------------------
Top Phrases that Compete with Other 'Peer' Phrases
------------------------------------------------------------------------
competing 它的 -4.2064 < 他的 -3.7473
competing 她的 -4.3475 < 他的 -3.7473
competing 曉得 -4.5578 < 小的 -4.4391
competing 看得 -4.5813 < 看的 -4.4522
competing 來得 -4.6760 < 來的 -3.6617
competing 不適 -4.6850 < 不是 -3.2992
competing 友人 -4.9188 < 有人 -3.8092
competing 步道 -4.9207 < 不到 -3.9628
competing 試用 -4.9367 < 是用 -4.2225
competing 不宜 -4.9639 < 不一 -4.0670
...and 2199 more entries suppressed

------------------------------------------------------------------------
Multi-Character Phrases with Issues but We Don't Care
------------------------------------------------------------------------
indifferent 過的 -4.3673 < 過+的 -4.3366 ㄍㄨㄛˋ-ㄉㄜ˙
indifferent 一一 -4.3851 < 一+一 -4.1643 ㄧ-ㄧ
indifferent 對的 -4.4410 < 對+的 -4.3133 ㄉㄨㄟˋ-ㄉㄜ˙
indifferent 不在 -4.4469 < 不+在 -4.4151 ㄅㄨˊ-ㄗㄞˋ
indifferent 看的 -4.4522 < 看+的 -4.4459 ㄎㄢˋ-ㄉㄜ˙
indifferent 方的 -4.4846 < 方+的 -4.4430 ㄈㄤ-ㄉㄜ˙
indifferent 會是 -4.4860 < 會+是 -4.4627 ㄏㄨㄟˋ-ㄕˋ
indifferent 開的 -4.6716 < 開+的 -4.4893 ㄎㄞ-ㄉㄜ˙
indifferent 想的 -4.6771 < 想+的 -4.5967 ㄒㄧㄤˇ-ㄉㄜ˙
indifferent 不大 -4.6816 < 不+大 -4.6284 ㄅㄨˊ-ㄉㄚˋ
...and 22924 more entries suppressed

------------------------------------------------------------------------
Emojis with No Covering Phrases (But May Have Smaller Covering Phrases)
------------------------------------------------------------------------
ㄎㄎ         ㄎㄜ-ㄎㄜ

[gemini-code-assist]

Code Review

This pull request integrates the language model analysis script into the curation package as curation.analyzers.find_cover_issues and introduces a utility module curation.utils.lmreader for reading LM files. It also updates the Makefile to include targets for formatting, typechecking, and running the analysis. The review feedback highlights three critical issues: a slicing bug when cutoff is set to -1 (unlimited), a copy-paste error using len(insufficients) instead of len(indifferents) in the indifferent section, and a potential IndexError along with a missing f-string interpolation in lmreader.py.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.