mbox

[0/5] Assume UTF-8 encoding for localedef input files

Message ID cover.1652994079.git.fweimer@redhat.com
Headers

Message

Florian Weimer May 19, 2022, 9:06 p.m. UTC 
  This is a backwards-compatible change because of two localedef bugs that
cause bytes outside the ASCII range to produce unpredictable results:

  If char is signed, conversion from the assumed ISO-8859-1 input format
  to a UCS-4 codepoint does not produce the correct result.

  If the output character set is not overlapping ISO-8859-1 in the
  characters used in the locale, the required character set conversion
  is not applied.

This is why I think we can switch to UTF-8 without impacting backwards
compatibility, and there is no need for an option to restore the old
behavior.

Tested on i686-linux-gnu and x86_64-linux-gnu.

Thanks,
Florian

Florian Weimer (5):
  locale: Turn ADDC and ADDS into functions in linereader.c
  locale: Fix signed char bug in lr_getc
  locale: Introduce translate_unicode_codepoint into linereader.c
  locale: localdef input files are now encoded in UTF-8
  de_DE: Convert to UTF-8

 NEWS                         |   4 +
 locale/programs/linereader.c | 504 ++++++++++++++++++++++-------------
 locale/programs/linereader.h |   2 +-
 localedata/locales/de_DE     |  32 +--
 4 files changed, 338 insertions(+), 204 deletions(-)


base-commit: 2d5ec6692f5746ccb11db60976a6481ef8e9d74f

Comments

Carlos O'Donell July 4, 2022, 7:54 p.m. UTC | #1 
On 5/19/22 17:06, Florian Weimer via Libc-alpha wrote:
> This is a backwards-compatible change because of two localedef bugs that
> cause bytes outside the ASCII range to produce unpredictable results:
> 
>   If char is signed, conversion from the assumed ISO-8859-1 input format
>   to a UCS-4 codepoint does not produce the correct result.
> 
>   If the output character set is not overlapping ISO-8859-1 in the
>   characters used in the locale, the required character set conversion
>   is not applied.
> 
> This is why I think we can switch to UTF-8 without impacting backwards
> compatibility, and there is no need for an option to restore the old
> behavior.

I can agree with that. In some sense I think the parsing of locale files is something
that can require developers and users to adjust the syntax, though we'd like for it
to be backwards compatible. In this case it couldn't have worked.

Thank you for working on this to make locales easier to use.

I particularly appreciate the example conversion of de_DE.

Overall the series looks good and we should commit this ahead of glibc 2.36 so we can
get any new strings translated for the TP project. This series particularly adds some
error messages for the use of UTF-8 in the locale sources.

Again, I really appreciate that this makes it easier for natural language speakers
to write, adjust, and review locale sources. In cases where disambiguation is required
we still have the capacity to write it differently if we need to. This continues the
early work to convert from U-codes to ASCII.

Just like last time we had this discussion the idea that glibc would support
compiling locale sources on a system that lacks UTF-8 is no longer a requirement
that we should have for the library.
 
> Tested on i686-linux-gnu and x86_64-linux-gnu.
> 
> Thanks,
> Florian
> 
> Florian Weimer (5):
>   locale: Turn ADDC and ADDS into functions in linereader.c
>   locale: Fix signed char bug in lr_getc
>   locale: Introduce translate_unicode_codepoint into linereader.c
>   locale: localdef input files are now encoded in UTF-8
>   de_DE: Convert to UTF-8
> 
>  NEWS                         |   4 +
>  locale/programs/linereader.c | 504 ++++++++++++++++++++++-------------
>  locale/programs/linereader.h |   2 +-
>  localedata/locales/de_DE     |  32 +--
>  4 files changed, 338 insertions(+), 204 deletions(-)
> 
> 
> base-commit: 2d5ec6692f5746ccb11db60976a6481ef8e9d74f