Add Emoji annotations with known pronunciations

[xatier]

This PR adds 3801+545=4346 emoji ennotation entries with SOT from unicode-org/cldr-json [1]. The file is also now sorted with LC_ALL=C sort -o Symbols.txt Symbols.txt.

This change is inspired by the emoji-mozc-import project [2]. I wrote a small script to collect the annotations from the SOT with known pronunciations from McBopomofo's dictionary. A more human-readable list of the additions can be found at [3].

import json
import collections
import string

# known phrases and pronunciations
with open("BPMFMappings.txt") as f:
    mapping = f.readlines()

known_phrases = collections.defaultdict(set)
for line in mapping:
    known_phrases[line.split()[0]].add("-".join(line.split()[1:]))

with open("Symbols.txt") as f:
    symbols = set(f.readlines())

# raw unicode annotation data
with open("annotations.json") as f:
    blob = json.load(f)

j = blob["annotations"]["annotations"]
phrases = collections.defaultdict(set)

for k in j:
    # skip RTL
    if "阿拉伯" in j[k]["default"]:
        continue

    for kk in j[k]["default"]:

        # skip ascii entries
        if any(c in kk for c in string.printable):
            continue

        # skip symbol entries
        if any(c in kk for c in ("×", "÷", "−", "，")):
            continue

        # skip lengthy entries
        if len(kk) > 8:
            continue

        phrases[kk].add(k)

for p in phrases:
    if p in known_phrases:
        for emoji in phrases[p]:
            for pronunciations in known_phrases[p]:
                output = f"{emoji} {pronunciations} -8\n"
                if output not in symbols:
                    print(output, end="")

[1] https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-annotations-full/annotations/zh-Hant/annotations.json
[2] https://gitlab.com/Ayanonymous/emoji-mozc-import/-/tree/main?ref_type=heads
[3] https://gist.github.com/xatier/d12e3b709c98c9c20ea59d72b8664a61

AI usage disclosure: this is 100% human-written code. :p

[gemini-code-assist]

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

[xatier]

I wrote another one to find pronunciations with compound words; see the human-readable list [1].

import json
import collections
import string
import itertools


def all_partitions(s):
    n = len(s)
    for cuts in itertools.product([0, 1], repeat=n - 1):
        result = []
        start = 0
        for i, c in enumerate(cuts):
            if c == 1:
                result.append(s[start : i + 1])
                start = i + 1
        result.append(s[start:])
        yield result


# known phrases and pronunciations
with open("BPMFMappings.txt") as f:
    mapping = f.readlines()

known_phrases = collections.defaultdict(set)
for line in mapping:
    known_phrases[line.split()[0]].add("-".join(line.split()[1:]))

with open("Symbols.txt") as f:
    symbols = set(f.readlines())

# raw unicode annotation data
with open("annotations.json") as f:
    blob = json.load(f)

j = blob["annotations"]["annotations"]
phrases = collections.defaultdict(set)

for k in j:
    # skip RTL
    if "阿拉伯" in j[k]["default"]:
        continue

    for kk in j[k]["default"]:

        # skip ascii entries
        if any(c in kk for c in string.printable):
            continue

        # skip symbol entries
        if any(c in kk for c in ("×", "÷", "−", "，")):
            continue

        # skip lengthy entries
        if len(kk) > 8:
            continue

        phrases[kk].add(k)

for p in phrases:
    # not found from known_phrases, try to partition the string
    if p not in known_phrases:
        for partitions in all_partitions(p):
            # we have every pieces in the known phrases
            if all(pp in known_phrases for pp in partitions):
                for emoji in phrases[p]:
                    for combo in itertools.product(
                        *[known_phrases[pp] for pp in partitions]
                    ):
                        # print(
                        #    f"{emoji} {'-'.join(list(combo))} {'--'.join(partitions)}"
                        # )
                        print(f"{emoji} {'-'.join(list(combo))} -8")

[1] https://gist.github.com/xatier/a2bd55a20f753bac351ac85a2d2baf42

AI usage disclosure: I suck with algorithms; the all_partitions function is from Copilot output. Other codes are handwritten.

[lukhnos]

GitHub PR review UI warned about bidirectional/hidden texts. Any chance some RTL codepoints still got in?

[xatier]

Thanks for raising the concern. I believe GitHub is showing this warning due to emoji's use of zero-width characters [1] and emoji variation selectors [2].

We can find them with

$ grep -P -n "[\x{200B}-\x{200F}\x{FE00}-\x{FE0F}]" Source/Data/Symbols.txt

In fact, it also shows the same warning in the current version on master as well: https://github.com/openvanilla/McBopomofo/blob/master/Source/Data/Symbols.txt

Regarding the bi-directional controls, we have none of them.

Verify with:

$ grep -P -n "[\x{202A}-\x{202E}]" Source/Data/Symbols.txt

[1] https://unicode-explorer.com/b/2000
[2] https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)

[lukhnos]

LGTM. Thanks for this addition!

Would it be too much to ask for one single, squashed commit for this PR?

[xatier]

Sure, I can do that. I intentionally separate the unsorted results for easier review.

[xatier]

Thanks for merging this change.

Would you mind updating the contribution guide wiki page to mention that now the Symbols.txt file also needs to be sorted? Thanks!

[lukhnos]

Thanks for merging this change.

Thank you for the contribution!

Would you mind updating the contribution guide wiki page to mention that now the Symbols.txt file also needs to be sorted? Thanks!

Good suggestion. Done: https://github.com/openvanilla/McBopomofo/wiki/詞庫開發說明#請確保詞庫原始檔的排序

[ChiahongHong]

@xatier 感謝加入這個功能～ 不過我今天使用下來發現有一些問題想要討論：

主要原因是 CLDR annotations 收錄的對應詞彙範圍實在太廣。如果我們不做篩選就全部加入，可能會造成候選 emoji 過多的情況。例如輸入 ㄉㄨㄥˋ ㄨˋ 時，假設我因為某種需求想要選字，選單會變得幾乎滑不完 XDD

emoji.mp4

以下是超過 10 的注音：

注音	個數
ㄉㄨㄥˋ-ㄨˋ	67
ㄩㄣˋ-ㄉㄨㄥˋ	29
ㄕˊ-ㄨˋ	26
ㄖㄣˊ-ㄨˋ	24
ㄕㄨㄟˇ-ㄍㄨㄛˇ	19
ㄈㄤ-ㄒㄧㄥˊ	18
ㄒㄧㄥˊ-ㄉㄨㄥˋ-ㄅㄨˊ-ㄅㄧㄢˋ	18
ㄢˋ-ㄋㄧㄡˇ	15
ㄅㄧㄠˇ-ㄑㄧㄥˊ	14
ㄊㄧㄢ-ㄑㄧˋ	14
ㄧㄣ-ㄩㄝˋ	14
ㄒㄧㄥ-ㄗㄨㄛˋ	13
ㄩㄢˊ-ㄒㄧㄥˊ	12
ㄒㄧㄠˋ-ㄌㄧㄢˇ	11
ㄨㄛˇ-ㄞˋ-ㄋㄧˇ	11
ㄩㄝˋ-ㄑㄧˋ	11
ㄏㄨㄛˋ-ㄅㄧˋ	10
ㄕㄨ-ㄘㄞˋ	10
ㄢˋ-ㄐㄧㄢˋ	10
ㄩㄝˋ-ㄌㄧㄤˋ	10

延伸的第二個問題是：以前小麥 emoji 的對應詞比較精準，所以不特別考慮 emoji 候選順序也沒差，但現在加入了比較模糊、廣義的對應詞後，如果仍然不考慮排序可能會讓使用者感到困惑。例如輸入 ㄒㄧㄠ ㄈㄤˊ ㄔㄜ 時，使用者通常會預期優先出現 🚒，但目前會先出現消防員 👨‍🚒，因為 CLDR 裡它也有對應到「消防車」這個標籤。

[xatier]

Thanks for raising the concern. I certainly agree that we may want to trim down phrases with high count. Perhaps removing cases like ㄉㄨㄥˋ-ㄨˋ and ㄩㄣˋ-ㄉㄨㄥˋ? I am thinking maybe 20 can be a good cut-off for this. @lukhnos @zonble, please let me know if we want to trim this down a bit, I can raise another PR for it.

On the semantics side, it would be a more tricky issue that CLDR's data can be ambiguous or even incorrect [1] occasionally (or have Hong-Kong style Traditional Chinese phrases). However, CLDR is already the most complete SoT I can find with unicode annotations though.

https://github.com/unicode-org/cldr-json/blob/a79b499916d486dca4b0f74fe423ea457705fdd9/cldr-json/cldr-annotations-full/annotations/zh-Hant/annotations.json#L10

[zonble]

動物這個 case 看起來的確比較極端。我在想或許可以保留幾個在前面，然後一堆更很罕用的符號，或許可以移到整個 candidate list 的最後面，像是 動物 + emoji[:5] + 動.... + emoji[5:] 之類的。

@lukhnos 怎麼看？

[lukhnos]

我們先 rollback 這個 PR 吧。

[xatier]

Sounds good, feel free to rollback this PR until we find a better solution.