[Components Locked] Errors on Weblate - Could not parse translation files

[beatrizmilz]

Hi!
Weblate locked some components on "The R Project for Statistical Computing" due to parsing errors (related to encoding).

Example of the error:

Weblate could not parse the translation files while updating the translations.
'utf-8' codec can't decode byte 0xe1 in position 53: invalid continuation byte

This are the components locked:

• base (R GUI)
• base (C files)
• Graphics (C Files)

[daroczig]

There are PO files in the repo that are not UTF-8 and Weblate seems to have an issue with parsing those. This was already causing problems when updating the Weblate components to use merge + --no-wrap, but managed to workaround by temporarily converting the PO files to UTF-8, and did not need @bastistician 's help after all with running msgconv on these files, but it seems we might not really avoid that.

I think in the past we overcame this by doing the encoding conversion only in Weblate, but this time it would be preferred to avoid any diversion from SVN, so if @bastistician could do this update, that would be higly appreciated 🙇 If rather not due to time or other constraints, we can do the conversion in git and pass back to R Core SVN in the next patch.

[bastistician]

This was a real mess. I just fixed corrupted non-ASCII characters in several PO files and converted some with mixed encodings and those with latin-1 to UTF-8. @daroczig : Could you please try again with the current trunk?

[daroczig]

Thank you very much, and sorry for the mess! Weblate pulled the changes, but I cannot see it in its git repo 🤐 Honestly not sure what's going on .. I see the updates in upstream, but not in the main branch after merge. I'll look into this more tomorrow.

[daroczig]

As we are supposed to be in a clean state, I've run (after backup) git reset --hard origin/trunk and then forced Weblate to reparse all PO files -- this way managed to catch up with trunk after all :phew:

But unfortunately, still seeing one issue at src/library/base/po/RGui-lt.po -- @bastistician, could you please take a look when you have a chance? This is the only file causes parsing issues in Weblate.

There are quite a few other files using non-UTF encoding as well, it seems:

$ grep -rnP --include="*.po" --include="*.pot" 'Content-Type: text/plain; charset=(?!UTF-8)' .

./src/library/grDevices/po/R-en_GB.po:11:"Content-Type: text/plain; charset=ISO-8859-1\n"
./src/library/grDevices/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/grDevices/po/R-grDevices.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/grDevices/po/grDevices.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/grDevices/po/en_GB.po:16:"Content-Type: text/plain; charset=ISO-8859-1\n"
./src/library/grDevices/po/R-ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/tcltk/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/tcltk/po/R-tcltk.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/tcltk/po/R-ru.po:19:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/tcltk/po/tcltk.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/grid/po/grid.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/grid/po/R-grid.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/grid/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/grid/po/R-ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/methods/po/R-methods.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/methods/po/methods.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/methods/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/methods/po/R-ru.po:19:"Content-Type: text/plain; charset=koi8-r\n"
./src/library/stats4/po/R-stats4.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/stats4/po/R-ru.po:19:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/compiler/po/R-compiler.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/base/po/RGui-ru.po:17:"Content-Type: text/plain; charset=koi8-r\n"
./src/library/base/po/RGui-zh_CN.po:16:"Content-Type: text/plain; charset=gb2312\n"
./src/library/base/po/R.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/base/po/R-base.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/base/po/RGui-lt.po:16:"Content-Type: text/plain; charset=CP1257\n"
./src/library/base/po/RGui.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/base/po/en_GB.po:16:"Content-Type: text/plain; charset=ISO-8859-1\n"
./src/library/base/po/R-ru.po:18:"Content-Type: text/plain; charset=koi8-r\n"
./src/library/base/po/RGui-zh_TW.po:17:"Content-Type: text/plain; charset=CP950\n"
./src/library/base/po/en.po:17:"Content-Type: text/plain; charset=ISO-8859-1\n"
./src/library/graphics/po/graphics.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/graphics/po/R-graphics.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/graphics/po/R-ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/splines/po/splines.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/splines/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/splines/po/R-splines.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/splines/po/R-ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/utils/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/utils/po/utils.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/utils/po/R-ru.po:19:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/utils/po/R-utils.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/stats/po/ru.po:19:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/stats/po/R-stats.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/stats/po/stats.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/stats/po/R-ru.po:19:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/parallel/po/R-parallel.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/parallel/po/parallel.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/parallel/po/ru.po:17:"Content-Type: text/plain; charset=koi8-r\n"
./src/library/parallel/po/R-ru.po:17:"Content-Type: text/plain; charset=koi8-r\n"
./src/library/tools/po/ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"
./src/library/tools/po/tools.pot:17:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/tools/po/R-tools.pot:11:"Content-Type: text/plain; charset=CHARSET\n"
./src/library/tools/po/R-ru.po:18:"Content-Type: text/plain; charset=KOI8-R\n"

But have not seen Weblate complain about these .. so probably non-UTF is fine, but mixed (messed up) encodings are less so 🤐