[v5] Locales: Cyrillic -> ASCII transliteration table [BZ #2872]

  Hi Rafal,

Thanks for the thorough checking, it really helps.

On 13.10.2018 02:59, Rafal Luzynski wrote:
> Technical issue:  Please either attach your patch to the email 
> message or paste it inline, not both.  The patch as it is now is not 
> applicable. I had to edit it manually to apply.
>> diff -uNr a/localedata/locales/C b/localedata/locales/C --- 
>> a/localedata/locales/C 2018-10-11 15:10:12.000000000 +0000 +++ 
>> b/localedata/locales/C 2018-10-11 15:10:43.000000000 +0000
> 
> There is no such file.  Where have you got the source code from?
> Are you sure this is glibc? :-)

I was running my patch process against the Ubuntu 18.04 version of
localedata/locales. Now I have checked out the GitHub glibc source v2.28
and done the same. Please find the new patch attached. I am not
submitting it as a patch request because we have not yet addressed the
rest of your comments below. But at least this should be working as a
patch for you. Please let me know if there is any problem there still.

>> [...] From this patch I have excluded locales that already mention 
>> cyrillic or have a transliteration table for it: az_AZ 
>> iso14651_t1_common ky_KG mn_MN sr_RS tg_TJ tk_TM tt_RU uk_UA uz_UZ 
>> uz_UZ@cyrillic
> 
> I confirm that these locales are excluded and there are no other 
> missing locales.

Because of the surprisingly different list of locales between Ubuntu and
glibc there is now a different list of excluded ones as well.

mn_MN
sr_RS
tg_TJ
tk_TM
tt_RU
uk_UA
uz_UZ
uz_UZ@cyrillic
uk_UA

az_AZ, ky_KG are now included because they don't have cyrillic translit
in glibc. iso14651_t1_common is still implicitly excluded, because it
doesn't have 'translit_end' string.

Somehow az_AZ and tr_TR from glibc fail to transliterate Cyrillic even
after the patch applied (az_AZ is explicitly including tr_TR). I do not
see a reason, maybe you could check?

> Regarding the tests, I think there is no complete transliteration 
> test suite at the moment.  Probably the only test is 
> localedata/bug-iconv-trans.c. You can also see the collation tests 
> placed in the same directory, they use those multiple *.UTF-8.in 
> files.
> 
> You can skip the tests for now.

In the copy of localedata/bug-iconv-trans.c lines 10-11 we could just
change the list of the symbols we are now transliterating

  const char str[] = "ÄäÖöÜüß";
  const char expected[] = "AEaeOEoeUEuess";

like this

  const char str[] =
"ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУУ́ФХЦЧШЩъЫьЭЮЯабвгдежзийклмнопрстуу́фхцчшщЪыЬэюяёђѓєѕіїјљњћќўџѪѫѲѳѴѵҌҍ
ҐґҒғҔҕҖҗҚқҞҟҢңҤҥҦҧҨҩҪҫҬҭҮүҲҳҴҵҺһҼҽҾҿӀӁӂӋӌӐӑӒӓӖӗӘәӜӝӞӟӠӡӤӥӦӧӨөӰӱӲӳӴӵӸӹ’"
  const char expected[] =
"YODJG`YEZ`IYIJL`N`TSHK`U`DHABVGDEZHZIJKLMNOPRSTUU`FXCZCHSHSHHA`Y``E`YUYAabvgdezhzijklmnoprstuu`fxczchsh
shh``y``e`yuyayodjg`yez`iyijl`n`tshk`u`dhO`o`FHfhYHyhE`e`G`g`GHghGHghZH`zh`K`k`K`k`N`n`NGngP`p`O`o`C`C`
T`t`UuH`h`TCZtczSH`SH`CH`ch`CH`ch`iZH`zh`CH`ch`A`a`A`a`E`e`A`a`ZH`zh`Z`z`Z`z`I`i`O`o`O`o`U`u`U`u`CH`ch`
Y`y`'";

First I though they could just be added but not all locales
transliterate Umlauts so just extending the current test won't do as it
will fail for those locales.

>> [...] diff -uNr a/localedata/locales/am_ET 
>> b/localedata/locales/am_ET --- a/localedata/locales/am_ET 
>> 2018-10-11 15:10:11.000000000 +0000 +++ b/localedata/locales/am_ET 
>> 2018-10-11 15:10:43.000000000 +0000 @@ -1394,6 +1394,7 @@ <U137A> 
>> <U0060><U0039><U0030> <U137B> <U0060><U0031><U0030><U0030> <U137C> 
>> <U0060><U0031><U0030><U0030><U0030><U0030> +include 
>> "translit_cyrillic";"" translit_end % END LC_CTYPE
> 
> Shouldn't “include "translit_cyrillic";""” be placed before the 
> custom rules, together with other includes?  The same in more files, 
> I will not mention them all.

If I recall correctly it is because of the
"translit_end
END LC_CTYPE"
part at the end of the translit_cyrillic. This way it works for any
locale, regardless whether it has translit itself or not. And being at
the end it does not supersede any previous transliteration that may be
there for a reason.

As with some other comments, I am not super familiar with the formats of
glibc files. So if you have a definitive suggestion - pls. formulate it
as an imperative, not a question.

>> [...] +translit_start + +% CYRILLIC CAPITAL LETTER IO +<U0401> 
>> <U00CB>;"<U0059><U004F>"
> 
> This says that for ASCII (GOST 7.79 System B) you would like to 
> transliterate "Ё" as "YO" but the table in Wikipedia says "Yo".  I 
> understand that one or another may be correct depending on the 
> context but we should be consistent and also better let's stick with 
> the standard.

The choice for YO, SH, YA, ZH etc. is to avoid naming collisions for
example for "Сх" and "Ш" that would both transliterate to Sh:
With SH:"Схема"->"Shema" but "Шема"->"SHema"
With Sh:"Схема"->"Shema" and "Шема"->"Shema". Collision!
This is important e.g. for renaming files, grouping as in using uniq etc.

> 
>> +% CYRILLIC CAPITAL LETTER DJE +<U0402> <U0110>;"<U0044><U004A>"
> 
> This says "DJ" but System B does not mention it.  Where does it come 
> from? Also, I think it should be "Dj" rather than "DJ".
I took the first two letters from its name.

>> [...] +% CYRILLIC UNDEFINED +<U0423><U0301> 
>> <U00DA>;"<U0055><U0060>"
> 
> 1. I think it should be named "CYRILLIC CAPITAL LETTER U WITH ACUTE".
> 2. OK, the System A table mentions this letter but System B does not.
> Somehow we should handle it.  I think that "U`" is the best we can do
> for now. 3. It must be tested whether this actually works.
1. Let's do it just before you are ready to commit the patch, because it
breaks formulas in my worksheet and I will have to do it manually?
3. I have tested and it doesn't work/gets ignored. But if you were to
handle COMBINING it would work, wouldn't it?

>> [...] +% CYRILLIC CAPITAL LETTER HA +<U0425> <U0048>;<U0058>
> 
> I don't think that "H" is unavailable in any encoding therefore it 
> will always be transliterated as "H" and never as "X".  We can't
> help it and I don't think it is bad.
> 
But we can keep this for when/if there is a way to explicitly request
transcription instead of transliteration.

>> +% CYRILLIC CAPITAL LETTER TSE +<U0426> <U0043>;"<U0043><U005A>"
> 
> 1. "CZ" - maybe should be "Cz"?> 2. Are we able to implement the
> rule: "c before i, e, y, j"?
> 
1. see for CYRILLIC CAPITAL LETTER IO
2. not sure what you are talking about in 2. but I believe it's not
possible as per Marko's email.

>> +% CYRILLIC CAPITAL LETTER HARD SIGN +<U042A> 
>> <U02BA>;"<U0041><U0060>"
> 
> "A`" is only for Bulgarian and should go to bg_BG.  How should we 
> transliterate an upper case hard sign to plain ASCII?  I think that 
> just "``", same as lower case.
This is to avoid collision. Besides AFAIK e.g. in Russian there is no
capital hard sign because there are no words starting with it.

> 
>> +% CYRILLIC CAPITAL LETTER YERU +<U042B> <U0059>;"<U0059><U0060>"
> 
> Again, as "Y" is always available it will never be transliterated as 
> "Y`".
> 
But we can keep this for when/if there is a way to explicitly request
transcription instead of transliteration.

Bests,
Diego

[v5] Locales: Cyrillic -> ASCII transliteration table [BZ #2872]

Commit Message

Comments

Patch