From patchwork Wed Feb 18 23:23:45 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alexandre Oliva X-Patchwork-Id: 5164 X-Patchwork-Delegate: carlos@redhat.com Received: (qmail 27007 invoked by alias); 18 Feb 2015 23:24:15 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 26990 invoked by uid 89); 18 Feb 2015 23:24:14 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.5 required=5.0 tests=AWL, BAYES_50, LIKELY_SPAM_SUBJECT, SPF_HELO_PASS, SPF_PASS, T_RP_MATCHES_RCVD autolearn=no version=3.3.2 X-HELO: mx1.redhat.com From: Alexandre Oliva To: "Carlos O'Donell" Cc: Pravin Satpute , Siddhesh Poyarekar , Mike FABIAN , libc-alpha@sourceware.org, Jens Petersen Subject: Re: [PATCH] [BZ 17588 13064] Update UTF-8 charmap and width to Unicode 7.0.0 References: <573624784.8871393.1416848051220.JavaMail.zimbra@redhat.com> <20141223111038.GA5172@spoyarek.pnq.redhat.com> <119234933.5523688.1422972847328.JavaMail.zimbra@redhat.com> <54E23EC9.5020400@redhat.com> Date: Wed, 18 Feb 2015 21:23:45 -0200 In-Reply-To: <54E23EC9.5020400@redhat.com> (Carlos O'Donell's message of "Mon, 16 Feb 2015 14:02:33 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 On Feb 16, 2015, "Carlos O'Donell" wrote: > On 02/12/2015 05:18 AM, Alexandre Oliva wrote: >>> Regression tested on x86_64-linux-gnu. Ok to install? > Yes, this version is OK to install if you fix all the nits. Thanks. > Despite complaints that a change in the generator would create > a smaller diff, that doesn't matter to me. The script changes were small and I figured it wouldn't hurt to merge them and reduce the diff, so I did. So I'll wait for another ACK before checking this in. I also added the downloaded files to the tree, so that binary distributors don't risk running afoul of the LGPL for lack of the .txt files. It's not clear that they would be required, but it doesn't hurt to put them in. I also added unicode-license.txt, copied from other packages that ship it. I couldn't find the text file for download from unicode.org, though I admittedly didn't search very thoroughly. > Nit: ChangeLog needs [BZ #xxx] etc. *check*. Heh, I didn't realize there were open bugs about this, in spite of the mention in the Subject. Doh! > Nit: This covers bugs 17588, 13064, *AND* 14094. *check* > Nit: Needs a NEWS entry describing this in full glory :-) * Character encoding and ctype tables were updated to Unicode 7.0.0, using new generator scripts contributed by Pravin Satpute and Mike FABIAN (Red Hat). These updates cause user visible changes, such as the fix for bug 17998. > Some might argue it fits better under "scripts" e.g. scripts/unicode-gen, > but I don't care. We can move it later if we think it should move at all. *nod* >>> * unicode-gen/gen_unicode_ctype.py: New generator. > Nit: Wrong copyright year e.g. 2014 -> 2015. *check*. I added ", 2015" after 2014 in the scripts. > Nit: We don't use "Contributed by" statements, they are instead pat of what > git records as Author or in the git commit message. *check*. I removed them from the scripts, and added them as "from" in the ChangeLog and in NEWS. I also removed the links that pointed to github as upstream, since I understand the GNU libc repository is going to hold the master copy, and the repository that was linked to is thus obsolescent. >>> * tst-ctype-de_DE.ISO-8859-1.in: Adjust, islower now returns >>> true for ordinal indicators. > Nit: This need a specific new BZ for the fix to user-visible behaviour. *check*: [BZ# 17998] Here's the header of the patch and the incremental changes to the scripts, from the previously posted version. The entire patch can be found in the lzip-compressed attachment. for localedata/ChangeLog [BZ #17588] [BZ #13064] [BZ #14094] [BZ #17998] * unicode-gen/Makefile: New. * unicode-gen/unicode-license.txt: New, from Unicode. * unicode-gen/UnicodeData.txt: New, from Unicode. * unicode-gen/DerivedCoreProperties.txt: New, from Unicode. * unicode-gen/EastAsianWidth.txt: New, from Unicode. * unicode-gen/gen_unicode_ctype.py: New generator, from Mike FABIAN . * unicode-gen/ctype_compatibility.py: New verifier, from Pravin Satpute and Mike FABIAN. * unicode-gen/ctype_compatibility_test_cases.py: New verifier module, from Mike FABIAN. * unicode-gen/utf8_gen.py: New generator, from Pravin Satpute and Mike FABIAN. * unicode-gen/utf8_compatibility.py: New verifier, from Pravin Satpute and Mike FABIAN. * charmaps/UTF-8: Update. * locales/i18n: Update. * gen-unicode-ctype.c: Remove. * tst-ctype-de_DE.ISO-8859-1.in: Adjust, islower now returns true for ordinal indicators. --- NEWS | 11 localedata/charmaps/UTF-8 |11946 ++++++--- localedata/gen-unicode-ctype.c | 784 - localedata/locales/i18n | 2652 +- localedata/tst-ctype-de_DE.ISO-8859-1.in | 2 localedata/unicode-gen/DerivedCoreProperties.txt |10794 ++++++++ localedata/unicode-gen/EastAsianWidth.txt | 2121 ++ localedata/unicode-gen/Makefile | 99 localedata/unicode-gen/UnicodeData.txt |27268 ++++++++++++++++++++ localedata/unicode-gen/ctype_compatibility.py | 546 .../unicode-gen/ctype_compatibility_test_cases.py | 951 + localedata/unicode-gen/gen_unicode_ctype.py | 751 + localedata/unicode-gen/unicode-license.txt | 50 localedata/unicode-gen/utf8_compatibility.py | 399 localedata/unicode-gen/utf8_gen.py | 286 15 files changed, 53278 insertions(+), 5382 deletions(-) delete mode 100644 localedata/gen-unicode-ctype.c create mode 100644 localedata/unicode-gen/DerivedCoreProperties.txt create mode 100644 localedata/unicode-gen/EastAsianWidth.txt create mode 100644 localedata/unicode-gen/Makefile create mode 100644 localedata/unicode-gen/UnicodeData.txt create mode 100755 localedata/unicode-gen/ctype_compatibility.py create mode 100644 localedata/unicode-gen/ctype_compatibility_test_cases.py create mode 100755 localedata/unicode-gen/gen_unicode_ctype.py create mode 100644 localedata/unicode-gen/unicode-license.txt create mode 100755 localedata/unicode-gen/utf8_compatibility.py create mode 100755 localedata/unicode-gen/utf8_gen.py diff --git a/NEWS b/NEWS index 0501d51..a59b68d 100644 --- a/NEWS +++ b/NEWS @@ -9,8 +9,15 @@ Version 2.22 * The following bugs are resolved with this release: - 4719, 15319, 15467, 15790, 16560, 17569, 17792, 17912, 17932, 17944, - 17949, 17964, 17965, 17967, 17969, 17978, 17987, 17991, 17996. + 4719, 13064, 14094, 15319, 15467, 15790, 16560, 17569, 17588, 17792, + 17912, 17932, 17944, 17949, 17964, 17965, 17967, 17969, 17978, 17987, + 17991, 17996, 17998. + +* Character encoding and ctype tables were updated to Unicode 7.0.0, using + new generator scripts contributed by Pravin Satpute and Mike FABIAN (Red + Hat). These updates cause user visible changes, such as the fix for bug + 17998. + Version 2.21 Incremental changes to the scripts: diff --git a/localedata/unicode-gen/ctype_compatibility.py b/localedata/unicode-gen/ctype_compatibility.py index 9535f81..19e9ee5 100755 --- a/localedata/unicode-gen/ctype_compatibility.py +++ b/localedata/unicode-gen/ctype_compatibility.py @@ -1,10 +1,7 @@ #!/usr/bin/python3 # -*- coding: utf-8 -*- -# Copyright (C) 2014 Free Software Foundation, Inc. +# Copyright (C) 2014, 2015 Free Software Foundation, Inc. # This file is part of the GNU C Library. -# Contributed by -# Pravin Satpute , 2014. -# Mike FABIAN , 2014. # # The GNU C Library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public diff --git a/localedata/unicode-gen/ctype_compatibility_test_cases.py b/localedata/unicode-gen/ctype_compatibility_test_cases.py index 09438d7..ab7f6dd 100644 --- a/localedata/unicode-gen/ctype_compatibility_test_cases.py +++ b/localedata/unicode-gen/ctype_compatibility_test_cases.py @@ -1,8 +1,6 @@ # -*- coding: utf-8 -*- -# Copyright (C) 2014 Free Software Foundation, Inc. +# Copyright (C) 2014, 2015 Free Software Foundation, Inc. # This file is part of the GNU C Library. -# Contributed by -# Mike FABIAN , 2014. # # The GNU C Library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public diff --git a/localedata/unicode-gen/gen_unicode_ctype.py b/localedata/unicode-gen/gen_unicode_ctype.py index 24155bd..559af79 100755 --- a/localedata/unicode-gen/gen_unicode_ctype.py +++ b/localedata/unicode-gen/gen_unicode_ctype.py @@ -1,9 +1,8 @@ #!/usr/bin/python3 # # Generate a Unicode conforming LC_CTYPE category from a UnicodeData file. -# Copyright (C) 2014 Free Software Foundation, Inc. +# Copyright (C) 2014, 2015 Free Software Foundation, Inc. # This file is part of the GNU C Library. -# Contributed by Mike FABIAN , 2014. # Based on gen-unicode-ctype.c by Bruno Haible , 2000. # # The GNU C Library is free software; you can redistribute it and/or diff --git a/localedata/unicode-gen/utf8_compatibility.py b/localedata/unicode-gen/utf8_compatibility.py index 4928e3e..e11327b 100755 --- a/localedata/unicode-gen/utf8_compatibility.py +++ b/localedata/unicode-gen/utf8_compatibility.py @@ -1,9 +1,7 @@ #!/usr/bin/python3 # -*- coding: utf-8 -*- -# Copyright (C) 2014 Free Software Foundation, Inc. +# Copyright (C) 2014, 2015 Free Software Foundation, Inc. # This file is part of the GNU C Library. -# Contributed by Pravin Satpute , 2014. -# Mike FABIAN , 2014 # # The GNU C Library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -27,8 +25,6 @@ To see how this script is used, call it with the “-h” option: $ ./utf8_compatibility.py -h … prints usage message … - -For issues upstream https://github.com/pravins/glibc-i18n ''' import sys diff --git a/localedata/unicode-gen/utf8_gen.py b/localedata/unicode-gen/utf8_gen.py index 9ffb7f6..670a628 100755 --- a/localedata/unicode-gen/utf8_gen.py +++ b/localedata/unicode-gen/utf8_gen.py @@ -1,10 +1,7 @@ #!/usr/bin/python3 # -*- coding: utf-8 -*- -# Copyright (C) 2014 Free Software Foundation, Inc. +# Copyright (C) 2014, 2015 Free Software Foundation, Inc. # This file is part of the GNU C Library. -# Contributed by -# Pravin Satpute and -# Mike Fabian - 2014 # # The GNU C Library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -28,13 +25,30 @@ from Unicode data. Usage: python3 utf8_gen.py UnicodeData.txt EastAsianWidth.txt It will output UTF-8 file - -For issues upstream https://github.com/pravins/glibc-i18n ''' import sys import re +# Auxiliary tables for Hangul syllable names, see the Unicode 3.0 book, +# sections 3.11 and 4.4. + +jamo_initial_short_name = [ + 'G', 'GG', 'N', 'D', 'DD', 'R', 'M', 'B', 'BB', 'S', 'SS', '', 'J', 'JJ', + 'C', 'K', 'T', 'P', 'H' +] + +jamo_medial_short_name = [ + 'A', 'AE', 'YA', 'YAE', 'EO', 'E', 'YEO', 'YE', 'O', 'WA', 'WAE', 'OE', + 'YO', 'U', 'WEO', 'WE', 'WI', 'YU', 'EU', 'YI', 'I' +] + +jamo_final_short_name = [ + '', 'G', 'GG', 'GS', 'N', 'NI', 'NH', 'D', 'L', 'LG', 'LM', 'LB', 'LS', + 'LT', 'LP', 'LH', 'M', 'B', 'BS', 'S', 'SS', 'NG', 'J', 'C', 'K', 'T', + 'P', 'H' +] + def ucs_symbol(code_point): '''Return the UCS symbol string for a Unicode character.''' if code_point < 0x10000: @@ -57,8 +71,15 @@ def process_range(start, end, outfile, name): # # So we expand the Hangul Syllables here: for i in range(int(start, 16), int(end, 16)+1 ): - outfile.write('{:s} {:s} {:s}\n'.format( - ucs_symbol(i), convert_to_hex(i), name)) + index2, index3 = divmod(i - 0xaC00, 28) + index1, index2 = divmod(index2, 21) + hangul_syllable_name = 'HANGUL SYLLABLE ' \ + + jamo_initial_short_name[index1] \ + + jamo_medial_short_name[index2] \ + + jamo_final_short_name[index3] + outfile.write('{:<11s} {:<12s} {:s}\n'.format( + ucs_symbol(i), convert_to_hex(i), + hangul_syllable_name)) return # UnicodeData.txt file has contains code point ranges like this: # @@ -73,13 +94,13 @@ def process_range(start, end, outfile, name): # .. /xe4/xb6/x80 for i in range(int(start, 16), int(end, 16), 64 ): if i > (int(end, 16)-64): - outfile.write('{:s}..{:s} {:s} {:s}\n'.format( + outfile.write('{:s}..{:s} {:<12s} {:s}\n'.format( ucs_symbol(i), ucs_symbol(int(end,16)), convert_to_hex(i), name)) break - outfile.write('{:s}..{:s} {:s} {:s}\n'.format( + outfile.write('{:s}..{:s} {:<12s} {:s}\n'.format( ucs_symbol(i), ucs_symbol(i+63), convert_to_hex(i), @@ -146,7 +167,7 @@ def process_charmap(flines, outfile): # the original UTF-8 file in glibc had them as # comments, so we keep these comment lines. outfile.write('%') - outfile.write('{:s} {:s} {:s}\n'.format( + outfile.write('{:<11s} {:<12s} {:s}\n'.format( ucs_symbol(int(fields[0], 16)), convert_to_hex(int(fields[0], 16)), fields[1]))