Use Unicode code points for country_isbn

  Hi,

make country_isbn definitions consistent across locales by using
Unicode code points not numerals everywhere. The code in
locale/categories.def and locale/programs/ld-address.c already
handles strings.

Please apply.

2015-06-05  Marko Myllynen  <myllynen@redhat.com>

	* li_BE: Add country_isbn.
	* li_NL: Likewise.

	* af_ZA: Use Unicode code point string for country_isbn.
	* ak_GH: Likewise.
	* bg_BG: Likewise.
	* cmn_TW: Likewise.
	* cy_GB: Likewise.
	* de_DE: Likewise.
	* en_NG: Likewise.
	* en_US: Likewise.
	* en_ZA: Likewise.
	* es_CR: Likewise.
	* es_US: Likewise.
	* fi_FI: Likewise.
	* fy_DE: Likewise.
	* gd_GB: Likewise.
	* ha_NG: Likewise.
	* hak_TW: Likewise.
	* hsb_DE: Likewise.
	* ht_HT: Likewise.
	* ia_FR: Likewise.
	* ig_NG: Likewise.
	* ka_GE: Likewise.
	* ku_TR: Likewise.
	* lb_LU: Likewise.
	* lzh_TW: Likewise.
	* mk_MK: Likewise.
	* mn_MN: Likewise.
	* nan_TW: Likewise.
	* nds_DE: Likewise.
	* nds_NL: Likewise.
	* oc_FR: Likewise.
	* pap_AN: Likewise.
	* ro_RO: Likewise.
	* sq_MK: Likewise.
	* sv_FI: Likewise.
	* tr_CY: Likewise.
	* tr_TR: Likewise.
	* uk_UA: Likewise.
	* unm_US: Likewise.
	* wa_BE: Likewise.
	* wae_CH: Likewise.
	* yi_US: Likewise.
	* yo_NG: Likewise.



Thanks,
  

On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> Hi,
> 
> make country_isbn definitions consistent across locales by using
> Unicode code points not numerals everywhere. The code in
> locale/categories.def and locale/programs/ld-address.c already
> handles strings.
> 
> Please apply.
> 
Possible but why, when these are numbers which are easier to read than
strings?
  

Hi,

On 2015-06-09 10:11, Ondřej Bílka wrote:
> On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
>>
>> make country_isbn definitions consistent across locales by using
>> Unicode code points not numerals everywhere. The code in
>> locale/categories.def and locale/programs/ld-address.c already
>> handles strings.
>>
>> Please apply.
>>
> Possible but why, when these are numbers which are easier to read than
> strings?

that's true, and I don't feel too strongly about this, but currently
some locales are using numbers and some are using Unicode code points so
there's a bit of inconsistency, also it's not that hard to read these
once one sees that e.g. 12 becomes "<U0031><U0032>" i.e. only the last
digit matters.

Thanks,
  

On 09 Jun 2015 13:12, Marko Myllynen wrote:
> On 2015-06-09 10:11, Ondřej Bílka wrote:
> > On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> >> make country_isbn definitions consistent across locales by using
> >> Unicode code points not numerals everywhere. The code in
> >> locale/categories.def and locale/programs/ld-address.c already
> >> handles strings.
> >>
> >> Please apply.
> >
> > Possible but why, when these are numbers which are easier to read than
> > strings?
> 
> that's true, and I don't feel too strongly about this, but currently
> some locales are using numbers and some are using Unicode code points so
> there's a bit of inconsistency, also it's not that hard to read these
> once one sees that e.g. 12 becomes "<U0031><U0032>" i.e. only the last
> digit matters.

i find many of the U markers pointlessly obscure, especially when they're used
for characters that are in the ASCII standard.  if we're standardizing on UTF8
encodings in general, why can't we convert these files as well ?  keep in mind
that i'm ignorant of the tooling around these files ;).
-mike
  

On Tue, Jul 21, 2015 at 04:18:40AM -0400, Mike Frysinger wrote:
> On 09 Jun 2015 13:12, Marko Myllynen wrote:
> > On 2015-06-09 10:11, Ond??ej Bílka wrote:
> > > On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> > >> make country_isbn definitions consistent across locales by using
> > >> Unicode code points not numerals everywhere. The code in
> > >> locale/categories.def and locale/programs/ld-address.c already
> > >> handles strings.
> > >>
> > >> Please apply.
> > >
> > > Possible but why, when these are numbers which are easier to read than
> > > strings?
> > 
> > that's true, and I don't feel too strongly about this, but currently
> > some locales are using numbers and some are using Unicode code points so
> > there's a bit of inconsistency, also it's not that hard to read these
> > once one sees that e.g. 12 becomes "<U0031><U0032>" i.e. only the last
> > digit matters.
> 
> i find many of the U markers pointlessly obscure, especially when they're used
> for characters that are in the ASCII standard.  if we're standardizing on UTF8
> encodings in general, why can't we convert these files as well ?  keep in mind
> that i'm ignorant of the tooling around these files ;).

The use of Unicode points helps making the locales portable, eg.
when crosscompiling for different architectures, including embedded systems, ebcdic
systems, utf-16 systems and utf8 systems, when you are on a different host platform.

For the ASCII characters one could use the symbolic character name from the
POSIX locale. They are much more readable than the Unicode code points, IMHO.

Best regards
Keld
  

On 07/21/2015 10:40 AM, keld@keldix.com wrote:

> The use of Unicode points helps making the locales portable, eg.
> when crosscompiling for different architectures, including embedded systems, ebcdic
> systems, utf-16 systems and utf8 systems, when you are on a different host platform.

Is this really a relevant use case?  Cross-compiling glibc to an EBCDIC
system?
  

On Tue, Jul 21, 2015 at 10:54:21AM +0200, Florian Weimer wrote:
> On 07/21/2015 10:40 AM, keld@keldix.com wrote:
> 
> > The use of Unicode points helps making the locales portable, eg.
> > when crosscompiling for different architectures, including embedded systems, ebcdic
> > systems, utf-16 systems and utf8 systems, when you are on a different host platform.
> 
> Is this really a relevant use case?  Cross-compiling glibc to an EBCDIC
> system?

I also mentioned other cases, which may be more relevant.. 

Best regards
keld
  

On 07/21/2015 11:02 AM, keld@keldix.com wrote:
> On Tue, Jul 21, 2015 at 10:54:21AM +0200, Florian Weimer wrote:
>> On 07/21/2015 10:40 AM, keld@keldix.com wrote:
>>
>>> The use of Unicode points helps making the locales portable, eg.
>>> when crosscompiling for different architectures, including embedded systems, ebcdic
>>> systems, utf-16 systems and utf8 systems, when you are on a different host platform.
>>
>> Is this really a relevant use case?  Cross-compiling glibc to an EBCDIC
>> system?
> 
> I also mentioned other cases, which may be more relevant.. 

I can't see how.  Unless someone maintains this code and processing
pipeline, it's not going to work with the current code.  Is anyone doing
it?  I doubt it.

We don't use trigraphs in C sources, so I really don't get why we have
to use an equivalent construct in the locale definitions.  Unless the
goal is to raise the bar for new contributors for some reason, but I
think the project has long walked away from that approach.
  

On 21 Jul 2015 10:40, keld@keldix.com wrote:
> On Tue, Jul 21, 2015 at 04:18:40AM -0400, Mike Frysinger wrote:
> > On 09 Jun 2015 13:12, Marko Myllynen wrote:
> > > On 2015-06-09 10:11, Ond??ej Bílka wrote:
> > > > On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> > > >> make country_isbn definitions consistent across locales by using
> > > >> Unicode code points not numerals everywhere. The code in
> > > >> locale/categories.def and locale/programs/ld-address.c already
> > > >> handles strings.
> > > >>
> > > >> Please apply.
> > > >
> > > > Possible but why, when these are numbers which are easier to read than
> > > > strings?
> > > 
> > > that's true, and I don't feel too strongly about this, but currently
> > > some locales are using numbers and some are using Unicode code points so
> > > there's a bit of inconsistency, also it's not that hard to read these
> > > once one sees that e.g. 12 becomes "<U0031><U0032>" i.e. only the last
> > > digit matters.
> > 
> > i find many of the U markers pointlessly obscure, especially when they're used
> > for characters that are in the ASCII standard.  if we're standardizing on UTF8
> > encodings in general, why can't we convert these files as well ?  keep in mind
> > that i'm ignorant of the tooling around these files ;).
> 
> The use of Unicode points helps making the locales portable, eg.
> when crosscompiling for different architectures, including embedded systems, ebcdic
> systems, utf-16 systems and utf8 systems, when you are on a different host platform.

i'm referring to the tools we use -- either inside of the source repo
(i.e. ones we wrote/maintain), or external ones that operate on our
files directly (i.e. gcc).  what actual problems do you see here ?
vague references like "cross-compiling is magic" aren't really that
interesting.

keep in mind we already use (and agreed to standardize on) UTF8 in
things like *.c and *.h and ChangeLog and READMEs and info pages.
-mike
  

On Tue, Jul 21, 2015 at 05:22:17AM -0400, Mike Frysinger wrote:
> On 21 Jul 2015 10:40, keld@keldix.com wrote:
> > On Tue, Jul 21, 2015 at 04:18:40AM -0400, Mike Frysinger wrote:
> > > On 09 Jun 2015 13:12, Marko Myllynen wrote:
> > > > On 2015-06-09 10:11, Ond??ej Bílka wrote:
> > > > > On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> > > > >> make country_isbn definitions consistent across locales by using
> > > > >> Unicode code points not numerals everywhere. The code in
> > > > >> locale/categories.def and locale/programs/ld-address.c already
> > > > >> handles strings.
> > > > >>
> > > > >> Please apply.
> > > > >
> > > > > Possible but why, when these are numbers which are easier to read than
> > > > > strings?
> > > > 
> > > > that's true, and I don't feel too strongly about this, but currently
> > > > some locales are using numbers and some are using Unicode code points so
> > > > there's a bit of inconsistency, also it's not that hard to read these
> > > > once one sees that e.g. 12 becomes "<U0031><U0032>" i.e. only the last
> > > > digit matters.
> > > 
> > > i find many of the U markers pointlessly obscure, especially when they're used
> > > for characters that are in the ASCII standard.  if we're standardizing on UTF8
> > > encodings in general, why can't we convert these files as well ?  keep in mind
> > > that i'm ignorant of the tooling around these files ;).
> > 
> > The use of Unicode points helps making the locales portable, eg.
> > when crosscompiling for different architectures, including embedded systems, ebcdic
> > systems, utf-16 systems and utf8 systems, when you are on a different host platform.
> 
> i'm referring to the tools we use -- either inside of the source repo
> (i.e. ones we wrote/maintain), or external ones that operate on our
> files directly (i.e. gcc).  what actual problems do you see here ?
> vague references like "cross-compiling is magic" aren't really that
> interesting.

It would mean that you cannot use the locale sources for crosscompiling when using
some different character sets on the hosting and the target machines.
Eg if you are making embedded systems on IOS or Windows or other utf16 machines
for an utf8 target, or making stuff for android. Or the other way round if you are
omn an utf8 host and generate locales for a utf16 target such as a utf16 embedded 
system or an iphone or ipad system.

I suggest you use the POSIX character names instead, eg 12 becomes "<1><2>"

> keep in mind we already use (and agreed to standardize on) UTF8 in
> things like *.c and *.h and ChangeLog and READMEs and info pages.

That is not related. Of cause we have our sources in a specific encoding,
and when sources are moved between platforms (aka portability) the 
sources text may be converted from one representation to another, 
which happens eg. when you move our sources to an IOS or Windows platform.

Best regards
Keld
  

On Tue, Jul 21, 2015 at 01:58:52PM +0200, Keld Simonsen wrote:
> I suggest you use the POSIX character names instead, eg 12 becomes "<1><2>"

I am sorry, in POSIX this would be "<one><two>" - I would then suggest that you
use the character naming of ISO TR 14652 or ISO TR 30112, which would be the
abovementioned "<1><2>"  -or the POSIX names, but the 14652/30112 names are more readable, IMHO.

Best regards
keld
  

On Tue, 21 Jul 2015, Keld Simonsen wrote:

> It would mean that you cannot use the locale sources for crosscompiling 
> when using some different character sets on the hosting and the target 
> machines. Eg if you are making embedded systems on IOS or Windows or 
> other utf16 machines for an utf8 target, or making stuff for android. Or 
> the other way round if you are omn an utf8 host and generate locales for 
> a utf16 target such as a utf16 embedded system or an iphone or ipad 
> system.

On the build system on which glibc is built, we can always assume that the 
glibc sources are the exact sequences of octets provided by the glibc 
project, not converted into another character set and without any 
conversions of line endings.  Furthermore, on any system using glibc and 
executing tools such as localedef with the installed locale source files, 
it can be assumed that those source files are the files shipped with 
glibc, not those files after conversion into another character set.  Use 
of glibc source files after conversion into another character set is 
outside the scope of the glibc project - glibc is not expected to build 
with such converted source files.

Now, it's true that the installed localedef utility should be usable in 
locale A to generate locale B, for any pair (A, B) of installed locales - 
rather than only being able to generate locales as part of the glibc build 
/ install process.  If localedef interprets locale sources in the 
character set of the locale in which it runs, that may mean the installed 
locale sources do need to be in ASCII.  How does localedef determine the 
character set in which to interpret the textual locale source files?
  

On Wed, Jul 22, 2015 at 05:25:04PM +0000, Joseph Myers wrote:
> On Tue, 21 Jul 2015, Keld Simonsen wrote:
> 
> > It would mean that you cannot use the locale sources for crosscompiling 
> > when using some different character sets on the hosting and the target 
> > machines. Eg if you are making embedded systems on IOS or Windows or 
> > other utf16 machines for an utf8 target, or making stuff for android. Or 
> > the other way round if you are omn an utf8 host and generate locales for 
> > a utf16 target such as a utf16 embedded system or an iphone or ipad 
> > system.
> 
> On the build system on which glibc is built, we can always assume that the 
> glibc sources are the exact sequences of octets provided by the glibc 
> project, not converted into another character set and without any 
> conversions of line endings.  Furthermore, on any system using glibc and 
> executing tools such as localedef with the installed locale source files, 
> it can be assumed that those source files are the files shipped with 
> glibc, not those files after conversion into another character set.  Use 
> of glibc source files after conversion into another character set is 
> outside the scope of the glibc project - glibc is not expected to build 
> with such converted source files.

Sounds strange. glibc is the library for the GNU C language. Standard ISO C
is coded character set independent, as is also POSIX. Why would the glibc project 
not follow ISO C and POSIX design goals? Why would glibc exclude itself
from Apple and Microsoft (utf16) and non-utf8 Linux and UNIX systems? 

Maybe we should clone glibc to make it available on other platforms
than those using utf8. Or maybe you are not correct. I have not been watching
the glibc project close enough to tell.

> Now, it's true that the installed localedef utility should be usable in 
> locale A to generate locale B, for any pair (A, B) of installed locales - 
> rather than only being able to generate locales as part of the glibc build 
> / install process.  If localedef interprets locale sources in the 
> character set of the locale in which it runs, that may mean the installed 
> locale sources do need to be in ASCII.  How does localedef determine the 
> character set in which to interpret the textual locale source files?

Yes, that is why we use UCS symbolic code points. I would then rather to be
fully consistent use UCS symbolic code points all the way thru a locale source,
it is a bit more cumbersome, but I would rather be consistent. And it would facilitate
the crosscompiling I wrote about. I don't think there is a mix of locales where it
matters on Linux boxes. Oh well, some thinkable scenarios:
Apple or Windosw users on a linux box, linux users on apple or Windows boxes,
Some mix with EBCDIC - more unlikely, but still thinkable is a big
mainfame and number cruncher environment, the mainframe being IBM mainframe
running VM/CMS and the number cruncher being a linux supercomputer, eg in
a financial institution.

Keld
  

On Wed, 22 Jul 2015, Keld Simonsen wrote:

> > On the build system on which glibc is built, we can always assume that the 
> > glibc sources are the exact sequences of octets provided by the glibc 
> > project, not converted into another character set and without any 
> > conversions of line endings.  Furthermore, on any system using glibc and 
> > executing tools such as localedef with the installed locale source files, 
> > it can be assumed that those source files are the files shipped with 
> > glibc, not those files after conversion into another character set.  Use 
> > of glibc source files after conversion into another character set is 
> > outside the scope of the glibc project - glibc is not expected to build 
> > with such converted source files.
> 
> Sounds strange. glibc is the library for the GNU C language. Standard 

No it's not.  It's the C library for the GNU system.  glibc has a range of 
requirements, including ELF, TLS, an MMU, two's complement integers, 
32-bit int, 32-bit or 64-bit long, 32-bit UTF-32 wchar_t, IEEE binary32 
float, IEEE binary64 double, various GNU tools present on the build system 
as documented in install.texi, ....

> ISO C is coded character set independent, as is also POSIX. Why would 
> the glibc project not follow ISO C and POSIX design goals? Why would 

Because glibc makes particular implementation choices in areas that are 
implementation-defined.  It's an implementation, not a meta-implementation 
that tries to cover the range of permitted implementation choices.  
Meta-implementations (at least of the language part of ISO C) exist, but 
they exist in the field of formal systems used to reason about C programs.

> glibc exclude itself from Apple and Microsoft (utf16) and non-utf8 Linux 
> and UNIX systems?

It's about 15-20 years since glibc was usable as a replacement C library 
for systems with an existing native non-free C library.  Those systems are 
not relevant to glibc nowadays (Apple and Microsoft systems fail the basic 
requirement of using ELF, which is assumed all over glibc).  UTF-16 is 
supported in iconv (only), just like EBCDIC.  Non-UTF-8 locales are 
supported, but deprecated (new non-UTF-8 locales should not be added, and 
any existing non-UTF-8 locales should have a UTF-8 counterpart), and to be 
usable in a POSIX-compliant way must have a character set that includes 
ASCII.

Given sufficiently many GNU tools built on a non-GNU build system, it 
should be possible to cross-compile glibc there - but localedef itself is 
only ever linked against glibc and run on a system using glibc (the 
cross-localedef functionality checked in to glibc is limited to allowing 
one glibc system to generate locales for another system with the same 
glibc version but a different endianness).

> > Now, it's true that the installed localedef utility should be usable in 
> > locale A to generate locale B, for any pair (A, B) of installed locales - 
> > rather than only being able to generate locales as part of the glibc build 
> > / install process.  If localedef interprets locale sources in the 
> > character set of the locale in which it runs, that may mean the installed 
> > locale sources do need to be in ASCII.  How does localedef determine the 
> > character set in which to interpret the textual locale source files?
> 
> Yes, that is why we use UCS symbolic code points. I would then rather to be

"Yes" does not answer my question about how localedef determines the 
character set of its input.

> fully consistent use UCS symbolic code points all the way thru a locale 
> source, it is a bit more cumbersome, but I would rather be consistent. 

I'd rather have some extension to allow a locale source file to declare 
that it is in UTF-8, and then use UTF-8 throughout except for control 
characters or combining characters used in isolation.
  

On Wed, Jul 22, 2015 at 08:02:23PM +0000, Joseph Myers wrote:
> > > Now, it's true that the installed localedef utility should be usable in 
> > > locale A to generate locale B, for any pair (A, B) of installed locales - 
> > > rather than only being able to generate locales as part of the glibc build 
> > > / install process.  If localedef interprets locale sources in the 
> > > character set of the locale in which it runs, that may mean the installed 
> > > locale sources do need to be in ASCII.  How does localedef determine the 
> > > character set in which to interpret the textual locale source files?
> > 
> > Yes, that is why we use UCS symbolic code points. I would then rather to be
> 
> "Yes" does not answer my question about how localedef determines the 
> character set of its input.
> 
> > fully consistent use UCS symbolic code points all the way thru a locale 
> > source, it is a bit more cumbersome, but I would rather be consistent. 
> 
> I'd rather have some extension to allow a locale source file to declare 
> that it is in UTF-8, and then use UTF-8 throughout except for control 
> characters or combining characters used in isolation.
>
I second that. It would be technically easy to do, so its mostly matter
of selecting proper interface. If we require some utf8 locale (if we
decide for C.UTF8 then use it otherwise for example en_US.

Then it would be matter of selecting different locale on files marked
say by having UTF8 in first line. Sample implementation would be:

fgets (first_line, 5, locale);
if (!memcmp (first_line, "UTF8", 4))
  setlocale(LC_ALL,"en_US.UTF8");
else
/* unget first line.  */
  

On 07/23/2015 06:27 PM, Ondřej Bílka wrote:
>> I'd rather have some extension to allow a locale source file to declare 
>> that it is in UTF-8, and then use UTF-8 throughout except for control 
>> characters or combining characters used in isolation.
>>
> I second that. It would be technically easy to do, so its mostly matter
> of selecting proper interface. If we require some utf8 locale (if we
> decide for C.UTF8 then use it otherwise for example en_US.
> 
> Then it would be matter of selecting different locale on files marked
> say by having UTF8 in first line. Sample implementation would be:
> 
> fgets (first_line, 5, locale);
> if (!memcmp (first_line, "UTF8", 4))
>   setlocale(LC_ALL,"en_US.UTF8");
> else
> /* unget first line.  */
> 

I agree with Joseph's position here.

Further to that, my primary goal is to make contribution for these
files easier.

I have no interest in the abstract cases that are not being supported
by anyone at the present moment.

Cheers,
Carlos.
  

On 06/09/2015 03:11 AM, Ondřej Bílka wrote:
> On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
>> Hi,
>>
>> make country_isbn definitions consistent across locales by using
>> Unicode code points not numerals everywhere. The code in
>> locale/categories.def and locale/programs/ld-address.c already
>> handles strings.
>>
>> Please apply.
>>
> Possible but why, when these are numbers which are easier to read than
> strings?
> 

I agree with Ondrej. Why?

The question we should all be asking ourselves here is:

	"What can we do to make it *easier* to maintain these files?"

Making everyone write in Unicode code points is not easier.

Joseph, Ondrej, and myself agree that we should find a way to just make
these files UTF-8. I expect that a precondition is going to be to add
an unremovable C.UTF-8 locale, which I think is important.

Cheers,
Carlos.
  

Carlos O'Donell wrote:
> Joseph, Ondrej, and myself agree that we should find a way to just make
> these files UTF-8. I expect that a precondition is going to be to add
> an unremovable C.UTF-8 locale, which I think is important.

I also like the idea of having these files be UTF-8.

Why is an unremovable C.UTF-8 locale a precondition, though?  We should be able 
to assume a properly-installed localedef and a minimum set of locales for 
running development tools like localedef.  The minimum set could include 
en_US.UTF-8.
  

On 07/23/2015 10:16 PM, Paul Eggert wrote:
> Carlos O'Donell wrote:
>> Joseph, Ondrej, and myself agree that we should find a way to just
>> make these files UTF-8. I expect that a precondition is going to be
>> to add an unremovable C.UTF-8 locale, which I think is important.
> 
> I also like the idea of having these files be UTF-8.
> 
> Why is an unremovable C.UTF-8 locale a precondition, though?  We
> should be able to assume a properly-installed localedef and a minimum
> set of locales for running development tools like localedef.  The
> minimum set could include en_US.UTF-8.

Agreed. I should not have said "precondition" when I really meant
"nice to have" since it simplifies some of the error handling
if you know you have a fallback UTF-8 locale you can use.

c.
  

On Thu, Jul 23, 2015 at 07:16:45PM -0700, Paul Eggert wrote:
> Carlos O'Donell wrote:
> >Joseph, Ondrej, and myself agree that we should find a way to just make
> >these files UTF-8. I expect that a precondition is going to be to add
> >an unremovable C.UTF-8 locale, which I think is important.
> 
> I also like the idea of having these files be UTF-8.
> 
> Why is an unremovable C.UTF-8 locale a precondition, though?  We
> should be able to assume a properly-installed localedef and a
> minimum set of locales for running development tools like localedef.
> The minimum set could include en_US.UTF-8.

You are then going to deviate from POSUX and ISO TR 30112 practice.

As you may know, I am involved in POSIX and 30112 standardization,
and I have tried to align 30112 with glibc practice. If you are deviating
 from POSIX guidelines, I have trouble unifying the two goals.

I also would like to have the 30112 standard implemented more broadly than just
glibc. Rumours has it that this is done somewhere else.
I would like to then use the glibc locales for the ISO 15897 registry,
and give maximum usability to those - that is the locales should be
fulle charset independent.

Also SC35 is looking at revising 30112 and in that revision we would
like to also update the character info to a new revision of 10646,
aligned to what is used in the sorting standard ISO 14651,
and I would like to use the glibc LC_CTYPE for that. 

I agree with that it should be easier to create locales.
One could do that with a GUI that helped create and proofread and test locales.

Best regards
Keld
  

On Thu, Jul 23, 2015 at 07:16:45PM -0700, Paul Eggert wrote:
> Carlos O'Donell wrote:
> >Joseph, Ondrej, and myself agree that we should find a way to just make
> >these files UTF-8. I expect that a precondition is going to be to add
> >an unremovable C.UTF-8 locale, which I think is important.
> 
> I also like the idea of having these files be UTF-8.
> 
> Why is an unremovable C.UTF-8 locale a precondition, though?  We
> should be able to assume a properly-installed localedef and a
> minimum set of locales for running development tools like localedef.
> The minimum set could include en_US.UTF-8.

I would recommend using the i18n locale - that is the purpose of i18n
locale to be the locale to buid all the other locales from.

Best regards
Keld
  

On Wed, Jul 22, 2015 at 08:02:23PM +0000, Joseph Myers wrote:
> On Wed, 22 Jul 2015, Keld Simonsen wrote:
> 
> > > On the build system on which glibc is built, we can always assume that the 
> > > glibc sources are the exact sequences of octets provided by the glibc 
> > > project, not converted into another character set and without any 
> > > conversions of line endings.  Furthermore, on any system using glibc and 
> > > executing tools such as localedef with the installed locale source files, 
> > > it can be assumed that those source files are the files shipped with 
> > > glibc, not those files after conversion into another character set.  Use 
> > > of glibc source files after conversion into another character set is 
> > > outside the scope of the glibc project - glibc is not expected to build 
> > > with such converted source files.
> > 
> > Sounds strange. glibc is the library for the GNU C language. Standard 
> 
> No it's not.  It's the C library for the GNU system.  glibc has a range of 
> requirements, including ELF, TLS, an MMU, two's complement integers, 
> 32-bit int, 32-bit or 64-bit long, 32-bit UTF-32 wchar_t, IEEE binary32 
> float, IEEE binary64 double, various GNU tools present on the build system 
> as documented in install.texi, ....

Yes, understood, but I don't think any of these requirements influenses the 
locales part.

> > ISO C is coded character set independent, as is also POSIX. Why would 
> > the glibc project not follow ISO C and POSIX design goals? Why would 
> 
> Because glibc makes particular implementation choices in areas that are 
> implementation-defined.  It's an implementation, not a meta-implementation 
> that tries to cover the range of permitted implementation choices.  
> Meta-implementations (at least of the language part of ISO C) exist, but 
> they exist in the field of formal systems used to reason about C programs.

I am also active in C standardization. I think it is a good goal to not
deviate and restrict an implementalton more than necessary. And at least 
not restrict it further than already implemented. That would lead to a loss
of functionality.


> > glibc exclude itself from Apple and Microsoft (utf16) and non-utf8 Linux 
> > and UNIX systems?
> 
> It's about 15-20 years since glibc was usable as a replacement C library 
> for systems with an existing native non-free C library.  Those systems are 
> not relevant to glibc nowadays (Apple and Microsoft systems fail the basic 
> requirement of using ELF, which is assumed all over glibc).  UTF-16 is 
> supported in iconv (only), just like EBCDIC.  Non-UTF-8 locales are 
> supported, but deprecated (new non-UTF-8 locales should not be added, and 
> any existing non-UTF-8 locales should have a UTF-8 counterpart), and to be 
> usable in a POSIX-compliant way must have a character set that includes 
> ASCII.

I thought cygwin was a GNU implementation for windows, and that it also
implemented glibc. I now understand that the cygwin libc is different from
glibc. But how different? Do they use glibc locales, or are they able to?

I would like the glibc locales to also be usable in other libc environments.
Most of all because they IMHO are the most comprehensive set of locales available.
So that would benefit users also outside glibc. Why not have this in mind
also for our project?

> Given sufficiently many GNU tools built on a non-GNU build system, it 
> should be possible to cross-compile glibc there - but localedef itself is 
> only ever linked against glibc and run on a system using glibc (the 
> cross-localedef functionality checked in to glibc is limited to allowing 
> one glibc system to generate locales for another system with the same 
> glibc version but a different endianness).
> 
> > > Now, it's true that the installed localedef utility should be usable in 
> > > locale A to generate locale B, for any pair (A, B) of installed locales - 
> > > rather than only being able to generate locales as part of the glibc build 
> > > / install process.  If localedef interprets locale sources in the 
> > > character set of the locale in which it runs, that may mean the installed 
> > > locale sources do need to be in ASCII.  How does localedef determine the 
> > > character set in which to interpret the textual locale source files?
> > 
> > Yes, that is why we use UCS symbolic code points. I would then rather to be
> 
> "Yes" does not answer my question about how localedef determines the 
> character set of its input.

My understanding is that the charset of the source is the charset of the locale
of the environment that localedef is running in. If the locale then is ASCII only
then there is no need for conversion of it - except for conversion 
into UTF16. Restricting the source further to invariant-ASCII also makes
the source portable to EBCDIC systems. Unicode restricts its sources to ASCII,
possibly also for this reason. Unicode do not publish their data in Unicode.

> > fully consistent use UCS symbolic code points all the way thru a locale 
> > source, it is a bit more cumbersome, but I would rather be consistent. 
> 
> I'd rather have some extension to allow a locale source file to declare 
> that it is in UTF-8, and then use UTF-8 throughout except for control 
> characters or combining characters used in isolation.

That would make it difficult to maintain in environments that is not using utf8.
Using ASCII only would make the locales maintainable on all systems.

Best regards
Keld
  

On Fri, 24 Jul 2015, Keld Simonsen wrote:

> > Because glibc makes particular implementation choices in areas that are 
> > implementation-defined.  It's an implementation, not a meta-implementation 
> > that tries to cover the range of permitted implementation choices.  
> > Meta-implementations (at least of the language part of ISO C) exist, but 
> > they exist in the field of formal systems used to reason about C programs.
> 
> I am also active in C standardization. I think it is a good goal to not
> deviate and restrict an implementalton more than necessary. And at least 
> not restrict it further than already implemented. That would lead to a loss
> of functionality.

The point of things being implementation-defined is to allow 
implementations flexibility in what is convenient for those 
implementations.  glibc duly uses that flexibility to adopt particular 
choices for implementation-defined behavior (some depending on the 
architecture, but most being globally fixed for all glibc configurations, 
so that all glibc code is free to rely on those choices).

> I thought cygwin was a GNU implementation for windows, and that it also
> implemented glibc. I now understand that the cygwin libc is different from
> glibc. But how different? Do they use glibc locales, or are they able to?

I don't think there's any use of glibc locales by newlib as Cygwin's libc.

> I would like the glibc locales to also be usable in other libc environments.
> Most of all because they IMHO are the most comprehensive set of locales available.
> So that would benefit users also outside glibc. Why not have this in mind
> also for our project?

I think CLDR is more likely to be the most comprehensive set of locales 
(it certainly claims to be "the largest and most extensive standard 
repository of locale data available"), and unlike glibc's locales is 
intended for wider use.  Even if we did want wider use for glibc's locales 
(beyond use by glibc's locale-dependent functions after having been 
compiled into binary form by glibc's localedef program from the same 
version of glibc) I think we should still say: UTF-8 is the way of the 
present and future, other multibyte character sets are legacy.  And, just 
as we require a range of GNU tools to build glibc, so we can rely on 
features of one part of the GNU system when working on another part, so we 
should require GNU localedef to build glibc's locales.

> > I'd rather have some extension to allow a locale source file to declare 
> > that it is in UTF-8, and then use UTF-8 throughout except for control 
> > characters or combining characters used in isolation.
> 
> That would make it difficult to maintain in environments that is not using utf8.

It would make the locales easier to maintain for people using UTF-8, the 
number of which (among people concerned with i18n) can be presumed to be 
much greater than the number using legacy character sets.
  

Keld Simonsen wrote:
> it should be easier to create locales.
> One could do that with a GUI that helped create and proofread and test locales.

I'm not aware of any such GUI, and even if one existed people would have to be 
trained to use it.  In contrast, we already have GUIs (e.g., Emacs) that people 
already know how to use and that work reasonably well with UTF-8 localedef sources.

Although the other goals you mention are laudable ones, surely they could be 
achieved by an automatic transformation of UTF-8 localedef sources into a 
less-readable equivalent with angle brackets, an equivalent that could be 
processed even by hypothetical tools operating in legacy multibyte locales. 
This shouldn't require a fancy GUI; it should be a relatively simple batch 
program.  Any engineering effort in this area would likely need this kind of 
transformation anyway, and any software developers in this specialized area 
should be able to take on this relatively minor extra task.
  

On Fri, Jul 24, 2015 at 09:50:46AM -0700, Paul Eggert wrote:
> Keld Simonsen wrote:
> >it should be easier to create locales.
> >One could do that with a GUI that helped create and proofread and test 
> >locales.
> 
> I'm not aware of any such GUI, and even if one existed people would have to 
> be trained to use it.  In contrast, we already have GUIs (e.g., Emacs) that 
> people already know how to use and that work reasonably well with UTF-8 
> localedef sources.
> 
> Although the other goals you mention are laudable ones, surely they could 
> be achieved by an automatic transformation of UTF-8 localedef sources into 
> a less-readable equivalent with angle brackets, an equivalent that could be 
> processed even by hypothetical tools operating in legacy multibyte locales. 
> This shouldn't require a fancy GUI; it should be a relatively simple batch 
> program.  Any engineering effort in this area would likely need this kind 
> of transformation anyway, and any software developers in this specialized 
> area should be able to take on this relatively minor extra task.

We could have a utility to do that, and probably there was one developed 
when Ulrich converted the mnemonic style to UCS codepoints. But maybe
that is lost. Or it could be part of the localedef utility,
given that localedef understands the full syntax of locales, then a conversion
option to and from different charsets and symbolic representations could be
done, with some better chances of being maintained and updated for
new features, and not lost.

I was also thinking of testing, how would a date be output with this date format?
I think there may be someting like that lying around, eg for KDE localization,
which I think is based on some other data and formats than glibc locales,
but it is a much bigger work than just doing some conversion of characters.

Keld
  

On Fri, Jul 24, 2015 at 03:11:15PM +0000, Joseph Myers wrote:
> On Fri, 24 Jul 2015, Keld Simonsen wrote:
> 
> > > Because glibc makes particular implementation choices in areas that are 
> > > implementation-defined.  It's an implementation, not a meta-implementation 
> > > that tries to cover the range of permitted implementation choices.  
> > > Meta-implementations (at least of the language part of ISO C) exist, but 
> > > they exist in the field of formal systems used to reason about C programs.
> > 
> > I am also active in C standardization. I think it is a good goal to not
> > deviate and restrict an implementalton more than necessary. And at least 
> > not restrict it further than already implemented. That would lead to a loss
> > of functionality.
> 
> The point of things being implementation-defined is to allow 
> implementations flexibility in what is convenient for those 
> implementations.  glibc duly uses that flexibility to adopt particular 
> choices for implementation-defined behavior (some depending on the 
> architecture, but most being globally fixed for all glibc configurations, 
> so that all glibc code is free to rely on those choices).

Yes, of cause implementation defined allowance is to be used.

I then have another hat on, as I am involved in writing the standards.
I have to have a generic point of view, and also from the users point of view
implementation defined items are no good for portability, so you
cannot be sure of your independence. You are bound to the implementation
of which you used the implementation defined specs.

I don't know about the goals of the glibc project, but there are a number
of possibilities to get out to a bigger audience. Actually the locales are mostly used
for end user apps, and glibc has a end user audience, that could be made bigger.
Eg both the Apple end user community and the Android user community
are way bigger than the glibc end user community. And they could be a target for
at least glibc locales. I believe both Apple and Google use POSIX derived localization,
including the locale model. I, at least as the editor of ISO TR 30122, need to have those
communities in sight. I have been cooperating with the glibc community, especially
Ulrich, but also with FSF as I have donated many locale and charmap specs to them.
And I am usig glibc i18n locale as the locale source in the standard. 

So I would welcome if glibc adhered to the design goals of character set independence,
that both POSIX and 30112 have, a design goal also shared by Unicode Inc.

> > I thought cygwin was a GNU implementation for windows, and that it also
> > implemented glibc. I now understand that the cygwin libc is different from
> > glibc. But how different? Do they use glibc locales, or are they able to?
> 
> I don't think there's any use of glibc locales by newlib as Cygwin's libc.

I believe if that is true, then they use something based on my earlier locales,
that I released to X/Open many years ago. Those were widely used in the industry,
as they were the only and most comprehensive locales around, freely available.
They also were the basis for many of the glibc locales. I think there is a potential
for glibc locales to take that position today.

> > I would like the glibc locales to also be usable in other libc environments.
> > Most of all because they IMHO are the most comprehensive set of locales available.
> > So that would benefit users also outside glibc. Why not have this in mind
> > also for our project?
> 
> I think CLDR is more likely to be the most comprehensive set of locales 
> (it certainly claims to be "the largest and most extensive standard 
> repository of locale data available"), and unlike glibc's locales is 
> intended for wider use.  Even if we did want wider use for glibc's locales 
> (beyond use by glibc's locale-dependent functions after having been 
> compiled into binary form by glibc's localedef program from the same 
> version of glibc) I think we should still say: UTF-8 is the way of the 
> present and future, other multibyte character sets are legacy.  And, just 
> as we require a range of GNU tools to build glibc, so we can rely on 
> features of one part of the GNU system when working on another part, so we 
> should require GNU localedef to build glibc's locales.

CLDR is not POSIX like locales, they are in XML. Also I believe they
are not in the same quality as the glibc locales. I for one had an experience with Unicode that
they would not take my specs, even if I represented Danish Standards. The result
was that their Danish spec did not adhere to Danish Standards and to Danish
official orthography rules.  I then gave up contact with them.

> > > I'd rather have some extension to allow a locale source file to declare 
> > > that it is in UTF-8, and then use UTF-8 throughout except for control 
> > > characters or combining characters used in isolation.
> > 
> > That would make it difficult to maintain in environments that is not using utf8.
> 
> It would make the locales easier to maintain for people using UTF-8, the 
> number of which (among people concerned with i18n) can be presumed to be 
> much greater than the number using legacy character sets.

Yes, but you are excluding some communities. So: easier for the majority,
impossible for a number of diverse minorities, which actually has the potential
to be much larger than the current user base.

Best regards
Keld
  

On Sat, 25 Jul 2015, Keld Simonsen wrote:

> > It would make the locales easier to maintain for people using UTF-8, the 
> > number of which (among people concerned with i18n) can be presumed to be 
> > much greater than the number using legacy character sets.
> 
> Yes, but you are excluding some communities. So: easier for the majority,
> impossible for a number of diverse minorities, which actually has the potential
> to be much larger than the current user base.

I think it's appropriate to say: if you want to use the glibc locales 
outside of glibc, you are responsible for maintaining the tools required 
to do so (e.g. for converting the encoding of locale source files).  I 
don't think such tools for conversion of encodings would be hard to write 
or need much maintenance when written (and in one direction - converting 
the ASCII files to UTF-8 - they might even be written by the glibc project 
as part of the initial conversion work).
  

Hi,

On 2015-07-24 03:23, Carlos O'Donell wrote:
> On 06/09/2015 03:11 AM, Ondřej Bílka wrote:
>> On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
>>>
>>> make country_isbn definitions consistent across locales by using
>>> Unicode code points not numerals everywhere. The code in
>>> locale/categories.def and locale/programs/ld-address.c already
>>> handles strings.
>>>
>> Possible but why, when these are numbers which are easier to read than
>> strings?
> 
> I agree with Ondrej. Why?

see above, for consistency.

> The question we should all be asking ourselves here is:
> 
> 	"What can we do to make it *easier* to maintain these files?"

Currently the definitions of this particular key across locales are
inconsistent and it doesn't make things easier as one can get confused
which form should be used for country_isbn.

> Making everyone write in Unicode code points is not easier.

The patch was only about making one individual key consistent, it's not
like this patch would add any additional generic burden.

Thanks,
  

On Mon, Aug 10, 2015 at 01:31:30PM +0300, Marko Myllynen wrote:
> Hi,
> 
> On 2015-07-24 03:23, Carlos O'Donell wrote:
> > On 06/09/2015 03:11 AM, Ond??ej Bílka wrote:
> >> On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
> >>>
> >>> make country_isbn definitions consistent across locales by using
> >>> Unicode code points not numerals everywhere. The code in
> >>> locale/categories.def and locale/programs/ld-address.c already
> >>> handles strings.
> >>>
> >> Possible but why, when these are numbers which are easier to read than
> >> strings?
> > 
> > I agree with Ondrej. Why?
> 
> see above, for consistency.
> 
> > The question we should all be asking ourselves here is:
> > 
> > 	"What can we do to make it *easier* to maintain these files?"
> 
> Currently the definitions of this particular key across locales are
> inconsistent and it doesn't make things easier as one can get confused
> which form should be used for country_isbn.
> 
> > Making everyone write in Unicode code points is not easier.
> 
> The patch was only about making one individual key consistent, it's not
> like this patch would add any additional generic burden.

Why not continue to use the UCS codepoints, as we do for all other strings in locales.
That would also add to consistency, and for portability (which I
understand is not a goal amongst glibc developers - but anyway...)

Best regards
keld
  

Hi,

On 2015-08-10 14:05, Keld Simonsen wrote:
> On Mon, Aug 10, 2015 at 01:31:30PM +0300, Marko Myllynen wrote:
>> On 2015-07-24 03:23, Carlos O'Donell wrote:
>>> On 06/09/2015 03:11 AM, Ond??ej Bílka wrote:
>>>> On Fri, Jun 05, 2015 at 05:57:06PM +0300, Marko Myllynen wrote:
>>>>>
>>>>> make country_isbn definitions consistent across locales by using
>>>>> Unicode code points not numerals everywhere. The code in
>>>>> locale/categories.def and locale/programs/ld-address.c already
>>>>> handles strings.
>>>>>
>>>> Possible but why, when these are numbers which are easier to read than
>>>> strings?
>>>
>>> I agree with Ondrej. Why?
>>
>> see above, for consistency.
>>
>>> The question we should all be asking ourselves here is:
>>>
>>> 	"What can we do to make it *easier* to maintain these files?"
>>
>> Currently the definitions of this particular key across locales are
>> inconsistent and it doesn't make things easier as one can get confused
>> which form should be used for country_isbn.
>>
>>> Making everyone write in Unicode code points is not easier.
>>
>> The patch was only about making one individual key consistent, it's not
>> like this patch would add any additional generic burden.
> 
> Why not continue to use the UCS codepoints, as we do for all other strings in locales.
> That would also add to consistency, and for portability (which I
> understand is not a goal amongst glibc developers - but anyway...)

that is exactly what my patch proposal was doing, nothing more, nothing
less: switch those locales using plain numbers for country_isbn to use
Unicode code points for country_isbn to make things consistent across
all locales.

In the long term we could look for alternatives for creating and
maintaining locales easier in general but in the short term I think the
best solution is to keep things consistent.

Thanks,