Incorporate transcription of accented <ý> to simple <y> in lemmas on the analysis side

[aarppe]

In order to more easily support using the same LEXC source for both crk and cwd (and other Western Cree dialects), we would want to incorporate a simple filter/transcriptor converting the accented <ý> to simple <y> to the analysis/underlying side of all the FSTs. For Woods Cree, we would similarly convert the accented <ý> to simple <th>. While the coding of <ý> is already implemented in the LEXC source on the stem side (for crk), this yet needs to be added to the lemma side of the LEXC code as well.

@snomos The REGEX source for the transcriptor has already been written and added to the repo as: https://github.com/giellalt/lang-crk/blob/main/src/filters/convert-accented-y-to-simple-y.regex - so what needs to be done is to incorporate this to the appropriate place(s) in the various Makefiles.

[snomos]

Can you give some example words to use with testing? What they look like in LexC, what the lemma should be, and what an expected word form should be.

[aarppe]

Comprehensive implementations of the LEXC files can now be found in lang-crk/src/fst/incoming, e.g. https://github.com/giellalt/lang-crk/blob/main/src/fst/incoming/stems/noun_stems.lexc

What I'm suggesting is that to the outside world, nothing changes, for now, so all the YAML tests should pass - it is only the innards that would be different.

[snomos]

Sure. I still would need to have an example to test with 😄

[aarppe]

All the final FSTs, e.g. the final normative FST analyzer (and the descriptive ones), should work as follows (so externally nothing will change, for now):

(1) akopayihow -> akopayihow+V+AI+Ind+3Sg

However, an interim step, before simplifying <ý>, would work as follows (with <ý> appearing in the lemma):

(2) akopayihow -> akopaýihow+V+AI+Ind+3Sg

I know how to transcribe things on the output/surface side, but not on the input/analysis side, so I was wondering where in the makefiles we add a simple filter/transcriptor regex [ y <- ý ] ; (which is already in the src/filters/convert-accented-y-to-simple-y.regex), that converts all cases of (2) to (1). The composition is clear (that's applied in src/morphological-fst-rules.mk), but I'm less certain about the correct insertion point on the GT side.

[snomos]

To recap: the idea is to have a common database for several Cree languages/dialects with abstract/historical forms as the common ground, and generate lexc for each language.

The question is really: what is the best place to do the conversion from <ý> to <y>? I would argue that it would be best to do the conversion already in the "export" to each language, ie in the generation of the language-specific lexc code. That would give:

• less fst maniputation (and with no actual benefit)
• ensure that lemma generation tests are still working (the tests extract lemmas from lexc, and would thus get <ý> in the lemma)

WDYT?

[aarppe]

What you're saying is what I originally thought, and that is what the LEXC exportation script currently does. However, not all contemporary <y> characters (in Plains), or <th> in Woods, are accented ones, so once the conversion is done, it cannot be done deterministically in reverse. In this respect, there could be a benefit in having some FSTs that keep the accented <ý>, as well as keeping that on the dictionary content side, and then having as an option, using transcriptors, whether one wants to see the accented <ý> or simple <y> or simple <th>. Otherwise, in order to be able to present the accented <ý> e.g. in itwêwina, we'd need to keep another field with the accented <ý> linked to the lemmas with the regular <y>.

It can be done either way.

[snomos]

So the display of the <ý> depends on the application/fst? It can not be assumed that all lemmas should always be <y>?

[aarppe]

Yes. Regular FSTs would have simple <y> and <th>, but dictionary FSTs would have <ý>. Typing <ý> is a challenge, so the regular end-products would use the simple <y>, whereas applications that could deal with <ý> easily could use that character internally.

This way, itwêwina could internally use accented <ý> in its storage of dictionary content which would then directly match with the dictionary FSTs also using <ý> - then, based on user settings (as a further variant on the selection of orthography), either the accented <ý> or simple <y> would be shown, with an itwêwina-internal transcriptor converting <ý> to <y> when so specified. Otherwise, we'd need to store dictionary entries with both alternative forms, and we couldn't easily generate the accented forms in the standardization of the search term or in paradigms.

However, elsewhere, such as in spell-checking and the standard FSTs that are generated, one should use the simple <y>, as it would be "cruel" to expect writers to type <ý>, and a meaningless complexification, since it serves no contemporary phonological distinction.