From patchwork Tue Feb 23 09:21:18 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Liebler X-Patchwork-Id: 11005 Received: (qmail 123244 invoked by alias); 23 Feb 2016 09:21:53 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 123124 invoked by uid 89); 23 Feb 2016 09:21:52 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.6 required=5.0 tests=AWL, BAYES_00, KAM_LAZY_DOMAIN_SECURITY, RP_MATCHES_RCVD autolearn=no version=3.3.2 spammy=BODY, 892, 7, 8927, 8928 X-HELO: e06smtp12.uk.ibm.com X-IBM-Helo: d06dlp02.portsmouth.uk.ibm.com X-IBM-MailFrom: stli@linux.vnet.ibm.com X-IBM-RcptTo: libc-alpha@sourceware.org From: Stefan Liebler To: libc-alpha@sourceware.org Cc: Stefan Liebler Subject: [PATCH 14/14] Fix UTF-16 surrogate handling. Date: Tue, 23 Feb 2016 10:21:18 +0100 Message-Id: <1456219278-5258-15-git-send-email-stli@linux.vnet.ibm.com> In-Reply-To: <1456219278-5258-1-git-send-email-stli@linux.vnet.ibm.com> References: <1456219278-5258-1-git-send-email-stli@linux.vnet.ibm.com> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16022309-0009-0000-0000-000007BB6068 According to the latest Unicode standard, a conversion from/to UTF-xx has to report an error if the character value is in range of an utf16 surrogate (0xd800..0xdfff). See https://sourceware.org/ml/libc-help/2015-12/msg00015.html. Thus this patch fixes this behaviour for converting from utf32 to internal and from internal to utf8. Furthermore the conversion from utf16 to internal does not report an error if the input-stream consists of two low-surrogate values. If an uint16_t value is in the range of 0xd800 .. 0xdfff, the next uint16_t value is checked, if it is in the range of a low surrogate (0xdc00 .. 0xdfff). Afterwards these two uint16_t values are interpreted as a high- and low-surrogates pair. But there is no test if the first uint16_t value is really in the range of a high-surrogate (0xd800 .. 0xdbff). If there would be two uint16_t values in the range of a low surrogate, then they will be treated as a valid high- and low-surrogates pair. This patch adds this test. ChangeLog: * iconvdata/utf-16.c (BODY): Report an error if first word is not a valid high surrogate. * iconvdata/utf-32.c (BODY): Report an error if the value is in range of an utf16 surrogate. * iconv/gconv_simple.c (BODY): Likewise. --- iconv/gconv_simple.c | 3 ++- iconvdata/utf-16.c | 12 ++++++++++++ iconvdata/utf-32.c | 2 +- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/iconv/gconv_simple.c b/iconv/gconv_simple.c index f66bf34..e5284e4 100644 --- a/iconv/gconv_simple.c +++ b/iconv/gconv_simple.c @@ -892,7 +892,8 @@ ucs4le_internal_loop_single (struct __gconv_step *step, if (__glibc_likely (wc < 0x80)) \ /* It's an one byte sequence. */ \ *outptr++ = (unsigned char) wc; \ - else if (__glibc_likely (wc <= 0x7fffffff)) \ + else if (__glibc_likely (wc <= 0x7fffffff \ + && (wc < 0xd800 || wc > 0xdfff))) \ { \ size_t step; \ unsigned char *start; \ diff --git a/iconvdata/utf-16.c b/iconvdata/utf-16.c index 2d74a13..dbbcd6d 100644 --- a/iconvdata/utf-16.c +++ b/iconvdata/utf-16.c @@ -295,6 +295,12 @@ gconv_end (struct __gconv_step *data) { \ uint16_t u2; \ \ + if (__glibc_unlikely (u1 >= 0xdc00)) \ + { \ + /* This is no valid first word for a surrogate. */ \ + STANDARD_FROM_LOOP_ERR_HANDLER (2); \ + } \ + \ /* It's a surrogate character. At least the first word says \ it is. */ \ if (__glibc_unlikely (inptr + 4 > inend)) \ @@ -329,6 +335,12 @@ gconv_end (struct __gconv_step *data) } \ else \ { \ + if (__glibc_unlikely (u1 >= 0xdc00)) \ + { \ + /* This is no valid first word for a surrogate. */ \ + STANDARD_FROM_LOOP_ERR_HANDLER (2); \ + } \ + \ /* It's a surrogate character. At least the first word says \ it is. */ \ if (__glibc_unlikely (inptr + 4 > inend)) \ diff --git a/iconvdata/utf-32.c b/iconvdata/utf-32.c index 0d6fe30..25f6fc6 100644 --- a/iconvdata/utf-32.c +++ b/iconvdata/utf-32.c @@ -239,7 +239,7 @@ gconv_end (struct __gconv_step *data) if (swap) \ u1 = bswap_32 (u1); \ \ - if (__glibc_unlikely (u1 >= 0x110000)) \ + if (__glibc_unlikely (u1 >= 0x110000 || (u1 >= 0xd800 && u1 < 0xe000))) \ { \ /* This is illegal. */ \ STANDARD_FROM_LOOP_ERR_HANDLER (4); \