BZ #19575: Clarify status of entries in GB 18030-2005.

  On 02/09/2016 03:55 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> On 02/08/2016 05:19 PM, Andreas Schwab wrote:
>>> "Carlos O'Donell" <carlos@redhat.com> writes:
>>>
>>>> This patch is only to clarify why these entries are being mapped
>>>> differently than in the original GB 18030-2005 standard.
>>>
>>> They aren't.
>>
>> Do you have a copy of the standard to verify that?
> 
> See charset/data/ucm/gb-18030-2005.ucm in ICU.

That's not a copy of the standard.

"CJKV Information Processing" by Dr. Ken Lunde on page 108
explicitly states that GB-18030-2005 has 24 PUA mappings
that with Unicode 4.1 or newer can be mapped to non-PUA
equivalents and he describes the 24 characters, and the ICU
ucm data does exactly that.

This does not match the published standard, but that is OK, 
it's best practice not to use PUA mappings if you can avoid
it when later Unicode versions include non-PUA equivalents
(as we do also in glibc).

All I want to clarify in the glibc version of these files
is that the data is not identical to the standard as published.

v2 of the patch follows.

OK to checkin?

Cheers,
Carlos.

2016-02-09  Carlos O'Donell  <carlos@redhat.com>

	[BZ #19575]
	* charmaps/GB18030: Document PUA to non-PUA equivalents.

---
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> 	* charmaps/GB18030: Document PUA to non-PUA equivalents.

This is by no way restricted to GB18030.  Several other legacy encodings
also have compatibility mappings that where remapped to official Unicode
points after they have become available.  The compatibility mappings are
usually included as comments in the code tables, as glibc can only use
the roudtrip code points.

If you want to add some verbiage, put it in a README.

Andreas.
  

On 02/09/2016 05:16 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> 	* charmaps/GB18030: Document PUA to non-PUA equivalents.
> 
> This is by no way restricted to GB18030.  Several other legacy encodings
> also have compatibility mappings that where remapped to official Unicode
> points after they have become available.  The compatibility mappings are
> usually included as comments in the code tables, as glibc can only use
> the roudtrip code points.

At this moment I'm only talking about GB 18030-2005.

> If you want to add some verbiage, put it in a README.

Why not in the file itself as the patch does? Keeping the notes right
beside the commented out mapping for developers to read if they have
any questions.

c.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> Why not in the file itself as the patch does?

Because it singles out GB18030 which only creates confusion.

Andreas.
  

On 02/09/2016 05:29 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> Why not in the file itself as the patch does?
> 
> Because it singles out GB18030 which only creates confusion.

How does it create confusion?

I find it adds clarity to the character map by explaining why
the comments are there e.g. non-normative changes which are
done because of a newer supporting Unicode with non-PUA code
points that can be used.

Cheers,
Carlos.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> the comments are there e.g. non-normative changes which are

Non-normative?

Andreas.
  

On 02/09/2016 09:26 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> the comments are there e.g. non-normative changes which are
> 
> Non-normative?

The glibc and ICU implementations of GB 18030-2005 have
non-normative mappings (not following the standard) for
the 24 PUA code point characters (as seen in my patch)
to their non-PUA equivalents.

Those mappings don't follow the printed normative standard
of GB 18030-2005, but that's OK, this is expected best practice.

Where "normative" means "part of the standard", and "non-normative"
means "not part of the standard."

Again, how does adding comments to one charmap create confusion?

Cheers,
Carlos.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> Those mappings don't follow the printed normative standard
> of GB 18030-2005, but that's OK, this is expected best practice.

# The 2005 version of gb18030 includes updates to previous mappings that use to map to PUA
# but are now mapped to actual Unicode codepoints.
# (CJKV 2nd edition)

Andreas.
  

On 02/09/2016 11:50 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> Those mappings don't follow the printed normative standard
>> of GB 18030-2005, but that's OK, this is expected best practice.
> 
> # The 2005 version of gb18030 includes updates to previous mappings that use to map to PUA
> # but are now mapped to actual Unicode codepoints.
> # (CJKV 2nd edition)

This statement is only partly correct. Some of the mappings were updated
but 24 mappings for PUA code points still remained.

CJKV Information Processing 2nd edition by Dr. Ken Lunde page 108:

"... Although PUA mappings for 24 characters are still printed in the GB
18030-2005 standard proper, it is important to note that as long as
Unicode Version 4.1 or greater is used, all 24 of these characters can be
safely mapped or otherwise handled without resorting to PUA code points ..."

Therefore the 24 mappings I mention in my comments are non-normative, they
are not part of the standard, but are a best-practice mapping to newer Unicode
environments.

My comments to the file clarify this situation.

Does that clarify my position?

Cheers,
Carlos.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> This statement is only partly correct. Some of the mappings were updated
> but 24 mappings for PUA code points still remained.

What are the updated mappings apart from the 24 being left?

Andreas.
  

On 02/10/2016 04:15 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> This statement is only partly correct. Some of the mappings were updated
>> but 24 mappings for PUA code points still remained.
> 
> What are the updated mappings apart from the 24 being left?

Sorry, I don't quite understand the question.

Could you please clarify exactly what you would like to know?

Cheers,
Carlos.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> On 02/10/2016 04:15 AM, Andreas Schwab wrote:
>> "Carlos O'Donell" <carlos@redhat.com> writes:
>> 
>>> This statement is only partly correct. Some of the mappings were updated
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> but 24 mappings for PUA code points still remained.
>> 
>> What are the updated mappings apart from the 24 being left?
>
> Sorry, I don't quite understand the question.
>
> Could you please clarify exactly what you would like to know?

Which are those updated mappings?

Andreas.
  

On 02/10/2016 09:14 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> On 02/10/2016 04:15 AM, Andreas Schwab wrote:
>>> "Carlos O'Donell" <carlos@redhat.com> writes:
>>>
>>>> This statement is only partly correct. Some of the mappings were updated
>                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> but 24 mappings for PUA code points still remained.
>>>
>>> What are the updated mappings apart from the 24 being left?
>>
>> Sorry, I don't quite understand the question.
>>
>> Could you please clarify exactly what you would like to know?
> 
> Which are those updated mappings?

So you would like to know which mappings changed between GB 18030-2000
and GB 18030-2005? I don't have such a list. In "CJKV Information Processing"
it is noted that there are 2 major areas of revision for 2000 -> 2005:

* Acknowledgment of CJK Unified Ideographs Extension B --- 42,711 hanzi

* Acknowledgment of the six regional scripts: Korean, Mongolian, Tai Le, Tibetan, Uyghur, and Yi.

So it supports all 42,711 hanzi characters, and the six scripts (all 4-byte
regions). There are also 4 pictoral glyph corrections.

May I ask why such a list of updated mappings is relevant here?

The only important thing here is that with those 24 PUA mappings made
into non-PUA equivalents the *entire* GB 18030-2005 can be represented
in Unicode without the use of PUA code points. Which is great because
it means normal unmodified programs can process and represent those
characters correctly.

In summary:
- glibc support GB 18030-2005.
- glibc modifies GB 18030-2005 to use 24 non-PUA code points and make
  the implementation fully use Unicode only.
- My comments are there to indicate the modifications for non-PUA code
  points (which deviate from the standard).

Cheers,
Carlos.
  

"Carlos O'Donell" <carlos@redhat.com> writes:

> May I ask why such a list of updated mappings is relevant here?

There are exactly 24 code point differences between -2000 and -2005.  Go
figure.

Andreas.
  

On 02/10/2016 10:57 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> May I ask why such a list of updated mappings is relevant here?
> 
> There are exactly 24 code point differences between -2000 and -2005.  Go
> figure.

I did not say that. There are only 24 PUA code points in -2005 which as of
Unicode 4.1 can be represented as non-PUA code points.

The differences between -2000 and -2005 are very large, as I outlined in
the rest of my response.

Do you have a sustained opposition to my additional documentation of the
non-normative mappings listed in the GB 18030-2005 charmap?

Cheers,
Carlos.