Skip to content

issue #63 include canonical language-code support with backward-compatible aliases #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
georgeslabreche wants to merge 1 commit into
base: master
Choose a base branch
from
Open

issue #63 include canonical language-code support with backward-compatible aliases #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 21 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,22 @@ Transliteration is the conversion of a text from one script to another. For inst

A citation would be much appreciated if you use CyrTranslit in a research publication:

[Georges Labrèche. (2025). CyrTranslit (1.2.0). Zenodo. https://doi.org/10.5281/zenodo.17663256](https://doi.org/10.5281/zenodo.17663256)
[Georges Labrèche. (2026). CyrTranslit (1.3.0). Zenodo. https://doi.org/10.5281/zenodo.17663256](https://doi.org/10.5281/zenodo.17663256)

BibTex entry:
```bibtex
@software{georges_labreche_nov2025,
@software{georges_labreche_jun2026,
author = {Georges Labrèche},
title = {CyrTranslit},
month = nov,
year = 2025,
month = jun,
year = 2026,
note = {{A Python package for bi-directional
transliteration of Cyrillic script to Latin script
and vice versa. Supports transliteration for Belarusian,
Bulgarian, Greek, Montenegrin, Macedonian, Mongolian,
Russian, Serbian, Tajik, and Ukrainian.}},
publisher = {Zenodo},
version = {1.2.0},
version = {1.3.0},
doi = {10.5281/zenodo.17663256},
url = {https://doi.org/10.5281/zenodo.17663256}
}
Expand Down Expand Up @@ -106,19 +106,21 @@ CyrTranslit is actively used as a reliable tool to advance research! Here's an i
CyrTranslit is [hosted in the Python Package Index (PyPI)](https://pypi.python.org/pypi/cyrtranslit) so it can be installed using pip:
```
python3 -m pip install cyrtranslit # latest version
python3 -m pip install cyrtranslit==1.2.0 # specific version
python3 -m pip install cyrtranslit>=1.2.0 # minimum version
python3 -m pip install cyrtranslit==1.3.0 # specific version
python3 -m pip install cyrtranslit>=1.3.0 # minimum version
```

## What languages are supported?

CyrTranslit currently supports bi-directional transliteration of Belarusian, Bulgarian, Greek, Montenegrin, Macedonian, Mongolian, Russian, Serbian, Tajik, and Ukrainian.

Language codes are based on ISO 639-1 standards. For Serbian, both `sr` (ISO 639-1 language code) and `rs` (ISO 3166-1 country code) are accepted:
Language codes use ISO 639 language identifiers: ISO 639-1 where available and ISO 639-3 for Montenegrin (`cnr`), which does not have an ISO 639-1 code. Historical country-code-like aliases (`by`, `me`, `rs`, `tj`, `ua`) and ISO 639 alpha-3 aliases are still accepted for backward compatibility:
```python
>>> import cyrtranslit
>>> cyrtranslit.supported()
['bg', 'by', 'el', 'me', 'mk', 'mn', 'rs', 'ru', 'sr', 'tj', 'ua']
['be', 'bg', 'cnr', 'el', 'mk', 'mn', 'ru', 'sr', 'tg', 'uk']
>>> cyrtranslit.supported(include_aliases=True)
['be', 'bel', 'bg', 'bul', 'by', 'cnr', 'el', 'ell', 'me', 'mk', 'mkd', 'mn', 'mon', 'rs', 'ru', 'rus', 'sr', 'srp', 'tg', 'tgk', 'tj', 'ua', 'uk', 'ukr']
```

## How do I use this?
Expand All @@ -131,9 +133,9 @@ CyrTranslit can be used both programatically and via command line interface.

```python
>>> import cyrtranslit
>>> cyrtranslit.to_latin("Прывітанне, свет!", "by")
>>> cyrtranslit.to_latin("Прывітанне, свет!", "be")
"Pryvitanne, svet!"
>>> cyrtranslit.to_cyrillic("Pryvitanne, svet!", "by")
>>> cyrtranslit.to_cyrillic("Pryvitanne, svet!", "be")
"Прывітанне, свет!"
```

Expand Down Expand Up @@ -161,9 +163,9 @@ CyrTranslit can be used both programatically and via command line interface.

```python
>>> import cyrtranslit
>>> cyrtranslit.to_latin("Република", "me")
>>> cyrtranslit.to_latin("Република", "cnr")
"Republika"
>>> cyrtranslit.to_cyrillic("Republika", "me")
>>> cyrtranslit.to_cyrillic("Republika", "cnr")
"Република"
```

Expand Down Expand Up @@ -211,19 +213,19 @@ CyrTranslit can be used both programatically and via command line interface.

```python
>>> import cyrtranslit
>>> cyrtranslit.to_latin("Ман мактуб навишта истодам", "tj")
>>> cyrtranslit.to_latin("Ман мактуб навишта истодам", "tg")
"Man maktub navišta istodam"
>>> cyrtranslit.to_cyrillic("Man maktub navišta istodam", "tj")
>>> cyrtranslit.to_cyrillic("Man maktub navišta istodam", "tg")
"Ман мактуб навишта истодам"
```

#### Ukrainian

```python
>>> import cyrtranslit
>>> cyrtranslit.to_latin("Під лежачий камінь вода не тече", "ua")
>>> cyrtranslit.to_latin("Під лежачий камінь вода не тече", "uk")
"Pid ležačyj kamin' voda ne teče"
>>> cyrtranslit.to_cyrillic("Pid ležačyj kamin' voda ne teče", "ua")
>>> cyrtranslit.to_cyrillic("Pid ležačyj kamin' voda ne teče", "uk")
"Під лежачий камінь вода не тече"
```

Expand Down Expand Up @@ -325,7 +327,7 @@ Try CyrTranslit by running it directly on the Python command line interface, e.g
```python
>>> import sys
>>> import cyrtranslit.cyrtranslit
>>> sys.argv.extend(['-l', 'UA'])
>>> sys.argv.extend(['-l', 'UK'])
>>> sys.argv.extend(['-i', 'tests/ua.txt'])
>>> sys.argv.extend(['-o', 'tests/output.txt'])
>>> cyrtranslit.cyrtranslit.main()
Expand Down Expand Up @@ -356,4 +358,4 @@ A big thank you to everyone who contributed:
- Tajik 🇹🇯: [@diejani](https://github.com/diejani).
- Ukrainian 🇺🇦: [@AnonymousVoice1](https://github.com/AnonymousVoice1).
- Mongolian 🇲🇳: [@Serbipunk](https://github.com/Serbipunk).
- Command Line Interface (CLI): [@ZJaume](https://github.com/ZJaume).
- Command Line Interface (CLI): [@ZJaume](https://github.com/ZJaume).
41 changes: 24 additions & 17 deletions cyrtranslit/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# -*- coding: utf-8 -*-
from .mapping import TRANSLIT_DICT
from .mapping import CANONICAL_LANG_CODES, TRANSLIT_DICT, normalize_lang_code
import sys

def __encode_utf8(_string):
Expand All @@ -23,24 +23,26 @@ def to_latin(string_to_transliterate, lang_code='sr', preserve_accents=False):
:return: A string of latin characters transliterated from the given cyrillic string.
'''

lang_code = normalize_lang_code(lang_code)

# First check if we support the cyrillic alphabet we want to transliterate to latin.
if lang_code.lower() not in TRANSLIT_DICT:
if lang_code not in TRANSLIT_DICT:
# If we don't support it, then just return the original string.
return string_to_transliterate

# If we do support it, check if the implementation is not missing before proceeding.
elif not TRANSLIT_DICT[lang_code.lower()]['tolatin']:
elif not TRANSLIT_DICT[lang_code]['tolatin']:
return string_to_transliterate

# Everything checks out, proceed with transliteration.
else:

# Get the character per character transliteration dictionary
transliteration_dict = TRANSLIT_DICT[lang_code.lower()]['tolatin'].copy()
transliteration_dict = TRANSLIT_DICT[lang_code]['tolatin'].copy()

# If preserve_accents=True and accented mappings exist, merge them (accented overrides standard)
if preserve_accents and 'tolatin_accented' in TRANSLIT_DICT[lang_code.lower()]:
transliteration_dict.update(TRANSLIT_DICT[lang_code.lower()]['tolatin_accented'])
if preserve_accents and 'tolatin_accented' in TRANSLIT_DICT[lang_code]:
transliteration_dict.update(TRANSLIT_DICT[lang_code]['tolatin_accented'])

# Initialize the output latin string variable
latinized_str = ''
Expand Down Expand Up @@ -74,22 +76,24 @@ def to_cyrillic(string_to_transliterate, lang_code='sr', preserve_accents=False)
:return: A string of cyrillic characters transliterated from the given latin string.
'''

lang_code = normalize_lang_code(lang_code)

# First check if we support the cyrillic alphabet we want to transliterate to latin.
if lang_code.lower() not in TRANSLIT_DICT:
if lang_code not in TRANSLIT_DICT:
# If we don't support it, then just return the original string.
return string_to_transliterate

# If we do support it, check if the implementation is not missing before proceeding.
elif not TRANSLIT_DICT[lang_code.lower()]['tocyrillic']:
elif not TRANSLIT_DICT[lang_code]['tocyrillic']:
return string_to_transliterate

else:
# Get the character per character transliteration dictionary
transliteration_dict = TRANSLIT_DICT[lang_code.lower()]['tocyrillic'].copy()
transliteration_dict = TRANSLIT_DICT[lang_code]['tocyrillic'].copy()

# If preserve_accents=True and accented mappings exist, merge them (accented overrides standard)
if preserve_accents and 'tocyrillic_accented' in TRANSLIT_DICT[lang_code.lower()]:
transliteration_dict.update(TRANSLIT_DICT[lang_code.lower()]['tocyrillic_accented'])
if preserve_accents and 'tocyrillic_accented' in TRANSLIT_DICT[lang_code]:
transliteration_dict.update(TRANSLIT_DICT[lang_code]['tocyrillic_accented'])

# Initialize the output cyrillic string variable
cyrillic_str = ''
Expand Down Expand Up @@ -142,11 +146,11 @@ def to_cyrillic(string_to_transliterate, lang_code='sr', preserve_accents=False)
(c in u'Zz' and c_plus_1 in u'Hh') or # z, zh
(c == u'\'' and c_plus_1 == u'\'') # ''
)) or \
(lang_code == 'ua' and (
(lang_code == 'uk' and (
(c in u'Jj' and c_plus_1 in u'eEaAuUiI') or # je, ja, ju
(c in u'Šš' and c_plus_1 in u'č') # šč
)) or \
(lang_code == 'by' and (
(lang_code == 'be' and (
(c in u'Jj' and c_plus_1 in u'uUaA') or # ju, ja
(c == u'\'' and c_plus_1 == u'\'') # '' for Ьь
)) or \
Expand All @@ -168,15 +172,15 @@ def to_cyrillic(string_to_transliterate, lang_code='sr', preserve_accents=False)
# In Bulgarian, the letter "щ" is represented by three latin letters: "sht",
# so we need this logic to support the third latin letter
if lang_code == 'bg' and \
index + 2 <= length_of_string_to_transliterate - 1 and \
index + 1 <= length_of_string_to_transliterate - 1 and \
(c == 'sh' or c == 'Sh' or c == 'SH') and \
string_to_transliterate[index + 1] in u'Tt':
index += 1
c += string_to_transliterate[index]

# Similarly in Russian, the letter "щ" шы represented by "shh".
if lang_code == 'ru' and \
index + 2 <= length_of_string_to_transliterate - 1 and \
index + 1 <= length_of_string_to_transliterate - 1 and \
(c == u'sh' or c == 'Sh' or c == 'SH') and \
string_to_transliterate[index + 1] in u'Hh': # shh
index += 1
Expand Down Expand Up @@ -212,8 +216,11 @@ def to_cyrillic(string_to_transliterate, lang_code='sr', preserve_accents=False)
return __encode_utf8(cyrillic_str)


def supported():
def supported(include_aliases=False):
''' Returns list of supported languages, sorted alphabetically.
:param include_aliases: If True, include legacy and ISO 639 alpha-3 aliases.
:return:
'''
return sorted(TRANSLIT_DICT.keys())
if include_aliases:
return sorted(TRANSLIT_DICT.keys())
return sorted(CANONICAL_LANG_CODES)
13 changes: 7 additions & 6 deletions cyrtranslit/cyrtranslit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
import cyrtranslit
from cyrtranslit.mapping import TRANSLIT_DICT
from cyrtranslit.mapping import TRANSLIT_DICT, normalize_lang_code
from argparse import ArgumentParser, FileType
import os
import sys

def __is_valid_language_code(parse, arg):
''' Validates inputted two-letter language code.
''' Validates inputted language code.
:param parse: The argument parser. Used to display error message.
:param arg: The language code argument.
'''
if arg.lower() not in TRANSLIT_DICT:
parser.error("The language code %s is not supported. Support language codes are: %s." % (arg, ", ".join(TRANSLIT_DICT.keys()).upper()))
lang_code = normalize_lang_code(arg)
if lang_code not in TRANSLIT_DICT:
parse.error("The language code %s is not supported. Supported language codes are: %s." % (arg, ", ".join(cyrtranslit.supported(include_aliases=True)).upper()))
else:
return arg
return lang_code

def main():
# Setup argument parser
Expand All @@ -33,7 +34,7 @@ def main():
# Language code for cyrillic text in inputted file.
# Required.
parser.add_argument("-l", dest="language_code", required=True,
help="two-letter ISO 639-1 language code of cyrillic text",
help="ISO 639 language code of cyrillic text",
type=lambda x: __is_valid_language_code(parser, x))

# Flag for reverse transliteration, i.e. from latin/roman alphabet to cyrillic.
Expand Down
Loading
Loading