RegionPreference: Align with UTS 35?

[anba]

RegionPreference doesn't seem to match UTS 35, see also https://unicode.org/reports/tr35/tr35.html#RegionOverride.

Issues:

1. CanonicalUnicodeSubdivision calls CanonicalizeUnicodeLocaleId to replace region aliases. It's unclear if this is valid per UTS 35. For example in Processing LocaleIds, unicode_subdivision_id replaces subdivisions using <subdivisionAlias> without first canonicalizing the subdivision's unicode_region_subtag part.

Example:

js> new Intl.Locale("en-u-rg-826zzzz").getHourCycles()[0]
"h12"
js> new Intl.Locale("en-u-rg-gbzzzz").getHourCycles()[0]
"h23"

Per the current spec text, both calls should return "h23". (826 is the numeric region for GB.)

2. Region Override defines that only "regular" and "unknown" regions are allowed and any other value should be ignored. Validity is checked against https://github.com/unicode-org/cldr/blob/main/common/validity/region.xml. I'm not sure if the "unknown" region should be accepted, though, because at least ICU4C seems to ignore it.

Examples:

js> new Intl.Locale("en-JP-u-rg-001zzzz").getCalendars()
["gregory", "japanese"]

The macro-region 001 has <calendarPreference> data, but the region override is ignored nonetheless.

js> new Intl.Locale("en-JP-u-rg-zzzzzz").getCalendars()
["gregory", "japanese"]
js> new Intl.Locale("ja-u-rg-zzzzzz-sd-thzzzz").getCalendars()
["buddhist", "gregory"]

Region override uses the "unknown" region ZZ, but it's ignored and instead the calendar preference from JP resp. TH is used.

js> new Intl.Locale("ja-u-sd-euzzzz").getCalendars()
["gregory", "japanese"]
js> new Intl.Locale("ja-u-sd-thzzzz").getCalendars()
["buddhist", "gregory"]

The subdivision using the macro-region EU is ignored and instead the calendar preference from JP (added through likely subtags) is used. When using a regular region (TH), the sd Unicode extension value is used.

---

Existing (browser) implementations directly call into ICU4C and don't implement any of the ECMA-402 specific operations, so changing the spec won't cause any web-compatibility issues. And ICU4C seems to ignore both region aliases and non-regular regions.

---

ICU4C is only buggy when using an empty Unicode extension value:

d8> new Intl.Locale("ja-u-rg").getCalendars()
["gregory", "islamic", "islamic-civil", "islamic-tbla"]

This probably happens because the empty string is treated as a shorthand for yes (either directly or by first replacing with the value true), and yes starts with the region code for YE (Yemen).

[ptomato]

On (1), this change doesn't make sense to me — what would be the benefit to not respect region aliases when they're part of a subdivision tag? If gb and 826 are aliases, and zzzz means "whole region", why should u-rg-gbzzzz and u-rg-826zzzz make the resulting locales different?

[sffc]

On (1), is the question whether -u-rg-826zzzz should be canonicalized to -u-rg-gbzzzz?

On (2), the -u-rg subtag should be applied first, as explained in https://unicode.org/reports/tr35/tr35.html#Region_Priority_Inheritance, which means that en-JP-u-rg-001zzzz should be returning the global preference list, not the Japan preference list. This makes sense: it is requesting English as spoken in Japan but with global preferences.

[sffc]

On a closer look, the region override in en-JP-u-rg-001zzzz is not valid according to https://unicode.org/reports/tr35/#RegionOverride because macroregions are not allowed in that position. So, that locale should be treated the same as en-JP.

[anba]

If gb and 826 are aliases, and zzzz means "whole region", why should u-rg-gbzzzz and u-rg-826zzzz make the resulting locales different?

This should probably be filed against UTS 35. If region aliases are supported in rg Unicode extension sequences, then I'd actually expect that these replacements happen during canonicalisation, i.e. new Intl.Locale("en-u-rg-826zzzz").toString() === "en-u-rg-gbzzzz"

[anba]

On a closer look, the region override in en-JP-u-rg-001zzzz is not valid according to https://unicode.org/reports/tr35/#RegionOverride because macroregions are not allowed in that position. So, that locale should be treated the same as en-JP.

I agree with that. The current spec text doesn't check the region kind, that's why #1059 adds this step to CanonicalUnicodeSubdivision:

If region is not a valid regular region code, return undefined.

[anba]

The examples in #1058 (comment) show the actual results in implementations to demonstrate that implementation don't match the current spec text. With #1059 applied, the spec should match how implementations work (except for bugs in ICU4C).

[ptomato]

I've created tc39/test262#5080 as a baseline expectation for how the region is selected, according to the current spec text.