preprocessor: -Wbidi-chars and UCNs [PR104030]

Message ID 20220120164439.42202-1-polacek@redhat.com
State New
Headers
Series preprocessor: -Wbidi-chars and UCNs [PR104030] |

Commit Message

Marek Polacek Jan. 20, 2022, 4:44 p.m. UTC
  Stephan Bergmann reported that our -Wbidi-chars breaks the build
of LibreOffice because we warn about UCNs even when their usage
is correct: LibreOffice constructs strings piecewise, as in:

  aText = u"\u202D" + aText;

and warning about that is overzealous.  Since no editor (AFAIK)
interprets UCNs to show them as Unicode characters, there's less
risk in misinterpreting them, and so perhaps we shouldn't warn
about them by default.  However, identifiers containing UCNs or
programs generating other programs could still cause confusion,
so I'm keeping the UCN checking.  To turn it on, you just need
to use -Wbidi-chars=unpaired,ucn or -Wbidi-chars=any,ucn.

The implementation is done by hardcoding and therefore ugly, but
my attempts to do something better quickly failed: this option is
marked as CPP, therefore needs Var and Init, and in turn Enum, etc.
And removing CPP doesn't sound like a great option.

Bootstrapped/regtested on x86_64-pc-linux-gnu, ok for trunk?

	PR preprocessor/104030

gcc/c-family/ChangeLog:

	* c.opt (Wbidi-chars): Also accept =any,ucn, =ucn,any,
	=unpaired,ucn, and =ucn,unpaired.

gcc/ChangeLog:

	* doc/invoke.texi: Update documentation for -Wbidi-chars.

libcpp/ChangeLog:

	* include/cpplib.h (enum cpp_bidirectional_level): Add
	bidirectional_unpaired_ucn and bidirectional_any_ucn enumerators.
	* lex.cc (maybe_warn_bidi_on_close): Don't warn about UCNs
	unless UCN checking is on.
	(maybe_warn_bidi_on_char): Likewise.

gcc/testsuite/ChangeLog:

	* c-c++-common/Wbidi-chars-10.c: Turn on UCN checking.
	* c-c++-common/Wbidi-chars-11.c: Likewise.
	* c-c++-common/Wbidi-chars-14.c: Likewise.
	* c-c++-common/Wbidi-chars-16.c: Likewise.
	* c-c++-common/Wbidi-chars-17.c: Likewise.
	* c-c++-common/Wbidi-chars-4.c: Likewise.
	* c-c++-common/Wbidi-chars-5.c: Likewise.
	* c-c++-common/Wbidi-chars-6.c: Likewise.
	* c-c++-common/Wbidi-chars-7.c: Likewise.
	* c-c++-common/Wbidi-chars-8.c: Likewise.
	* c-c++-common/Wbidi-chars-9.c: Likewise.
	* c-c++-common/Wbidi-chars-ranges.c: Likewise.
	* c-c++-common/Wbidi-chars-18.c: New test.
	* c-c++-common/Wbidi-chars-19.c: New test.
	* c-c++-common/Wbidi-chars-20.c: New test.
	* c-c++-common/Wbidi-chars-21.c: New test.
---
 gcc/c-family/c.opt                              | 14 +++++++++++++-
 gcc/doc/invoke.texi                             |  8 ++++++--
 gcc/testsuite/c-c++-common/Wbidi-chars-10.c     |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-11.c     |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-14.c     |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-16.c     |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-17.c     |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-18.c     | 11 +++++++++++
 gcc/testsuite/c-c++-common/Wbidi-chars-19.c     | 11 +++++++++++
 gcc/testsuite/c-c++-common/Wbidi-chars-20.c     | 11 +++++++++++
 gcc/testsuite/c-c++-common/Wbidi-chars-21.c     | 11 +++++++++++
 gcc/testsuite/c-c++-common/Wbidi-chars-4.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-5.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-6.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-7.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-8.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-9.c      |  2 +-
 gcc/testsuite/c-c++-common/Wbidi-chars-ranges.c |  2 +-
 libcpp/include/cpplib.h                         |  5 ++++-
 libcpp/lex.cc                                   | 14 +++++++++-----
 20 files changed, 88 insertions(+), 21 deletions(-)
 create mode 100644 gcc/testsuite/c-c++-common/Wbidi-chars-18.c
 create mode 100644 gcc/testsuite/c-c++-common/Wbidi-chars-19.c
 create mode 100644 gcc/testsuite/c-c++-common/Wbidi-chars-20.c
 create mode 100644 gcc/testsuite/c-c++-common/Wbidi-chars-21.c


base-commit: 5c12507f5d0bc080e4f346af99824e039236e61c
  

Patch

diff --git a/gcc/c-family/c.opt b/gcc/c-family/c.opt
index db65c14a7a5..f829656fc36 100644
--- a/gcc/c-family/c.opt
+++ b/gcc/c-family/c.opt
@@ -380,7 +380,7 @@  C ObjC C++ ObjC++ Warning Alias(Wbidi-chars=,any,none)
 
 Wbidi-chars=
 C ObjC C++ ObjC++ RejectNegative Joined Warning CPP(cpp_warn_bidirectional) CppReason(CPP_W_BIDIRECTIONAL) Var(warn_bidirectional) Init(bidirectional_unpaired) Enum(cpp_bidirectional_level)
--Wbidi-chars=[none|unpaired|any] Warn about UTF-8 bidirectional control characters.
+-Wbidi-chars=[none|unpaired|any|unpaired,ucn|any,ucn] Warn about UTF-8 bidirectional control characters.
 
 ; Required for these enum values.
 SourceInclude
@@ -398,6 +398,18 @@  Enum(cpp_bidirectional_level) String(unpaired) Value(bidirectional_unpaired)
 EnumValue
 Enum(cpp_bidirectional_level) String(any) Value(bidirectional_any)
 
+EnumValue
+Enum(cpp_bidirectional_level) String(any,ucn) Value(bidirectional_any_ucn)
+
+EnumValue
+Enum(cpp_bidirectional_level) String(ucn,any) Value(bidirectional_any_ucn)
+
+EnumValue
+Enum(cpp_bidirectional_level) String(unpaired,ucn) Value(bidirectional_unpaired_ucn)
+
+EnumValue
+Enum(cpp_bidirectional_level) String(ucn,unpaired) Value(bidirectional_unpaired_ucn)
+
 Wbool-compare
 C ObjC C++ ObjC++ Var(warn_bool_compare) Warning LangEnabledBy(C ObjC C++ ObjC++,Wall)
 Warn about boolean expression compared with an integer value different from true/false.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 67693d6c5cf..5498d908f18 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -328,7 +328,7 @@  Objective-C and Objective-C++ Dialects}.
 -Warray-bounds  -Warray-bounds=@var{n}  -Warray-compare @gol
 -Wno-attributes  -Wattribute-alias=@var{n} -Wno-attribute-alias @gol
 -Wno-attribute-warning  @gol
--Wbidi-chars=@r{[}none@r{|}unpaired@r{|}any@r{]} @gol
+-Wbidi-chars=@r{[}none@r{|}unpaired@r{|}any@r{|}unpaired,ucn@r{|}any,ucn@r{]} @gol
 -Wbool-compare  -Wbool-operation @gol
 -Wno-builtin-declaration-mismatch @gol
 -Wno-builtin-macro-redefined  -Wc90-c99-compat  -Wc99-c11-compat @gol
@@ -7802,7 +7802,7 @@  Attributes considered include @code{alloc_align}, @code{alloc_size},
 This is the default.  You can disable these warnings with either
 @option{-Wno-attribute-alias} or @option{-Wattribute-alias=0}.
 
-@item -Wbidi-chars=@r{[}none@r{|}unpaired@r{|}any@r{]}
+@item -Wbidi-chars=@r{[}none@r{|}unpaired@r{|}any@r{|}unpaired,ucn@r{|}any,ucn@r{]}
 @opindex Wbidi-chars=
 @opindex Wbidi-chars
 @opindex Wno-bidi-chars
@@ -7819,6 +7819,10 @@  bidi contexts.  @option{-Wbidi-chars=none} turns the warning off.
 @option{-Wbidi-chars=any} warns about any use of bidirectional control
 characters.
 
+By default, this warning does not warn about UCNs.  It is, however, possible
+to turn on such checking by using @option{-Wbidi-chars=unpaired,ucn} or
+@option{-Wbidi-chars=any,ucn}.
+
 @item -Wbool-compare
 @opindex Wno-bool-compare
 @opindex Wbool-compare
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-10.c b/gcc/testsuite/c-c++-common/Wbidi-chars-10.c
index 3f851b69e65..cdcdce2be08 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-10.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-10.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
 /* More nesting testing.  */
 
 /* RLE‫ LRI⁦ PDF‬ PDI⁩*/
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-11.c b/gcc/testsuite/c-c++-common/Wbidi-chars-11.c
index 270ce2368a9..ea83029d6b9 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-11.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-11.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
 /* Test that we warn when mixing UCN and UTF-8.  */
 
 int LRE_‪_PDF_\u202c;
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-14.c b/gcc/testsuite/c-c++-common/Wbidi-chars-14.c
index ba5f75d9553..cb6b05efac1 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-14.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-14.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
 /* Test PDI handling, which also pops any subsequent LREs, RLEs, LROs,
    or RLOs.  */
 
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-16.c b/gcc/testsuite/c-c++-common/Wbidi-chars-16.c
index baa0159861c..eaf0ec9a777 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-16.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-16.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=any" } */
+/* { dg-options "-Wbidi-chars=any,ucn" } */
 /* Test LTR/RTL chars.  */
 
 /* LTR<‎> */
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-17.c b/gcc/testsuite/c-c++-common/Wbidi-chars-17.c
index 07cb4321f96..341922146a7 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-17.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-17.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
 /* Test LTR/RTL chars.  */
 
 /* LTR<‎> */
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-18.c b/gcc/testsuite/c-c++-common/Wbidi-chars-18.c
new file mode 100644
index 00000000000..ae586d5e08c
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-18.c
@@ -0,0 +1,11 @@ 
+/* PR preprocessor/104030 */
+/* { dg-do compile } */
+/* By default, don't warn about UCNs.  */
+
+const char *
+fn ()
+{
+  const char *aText = "\u202D" "abc";
+/* { dg-bogus "unpaired" "" { target *-*-* } .-1 } */
+  return aText;
+}
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-19.c b/gcc/testsuite/c-c++-common/Wbidi-chars-19.c
new file mode 100644
index 00000000000..9985c3be7a5
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-19.c
@@ -0,0 +1,11 @@ 
+/* PR preprocessor/104030 */
+/* { dg-do compile } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
+
+const char *
+fn ()
+{
+  const char *aText = "\u202D" "abc";
+/* { dg-warning "unpaired" "" { target *-*-* } .-1 } */
+  return aText;
+}
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-20.c b/gcc/testsuite/c-c++-common/Wbidi-chars-20.c
new file mode 100644
index 00000000000..859f3d53779
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-20.c
@@ -0,0 +1,11 @@ 
+/* PR preprocessor/104030 */
+/* { dg-do compile } */
+/* { dg-options "-Wbidi-chars=any" } */
+
+const char *
+fn ()
+{
+  const char *aText = "\u202D" "abc";
+/* { dg-bogus "U\\+202D" "" { target *-*-* } .-1 } */
+  return aText;
+}
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-21.c b/gcc/testsuite/c-c++-common/Wbidi-chars-21.c
new file mode 100644
index 00000000000..2720b8a883e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-21.c
@@ -0,0 +1,11 @@ 
+/* PR preprocessor/104030 */
+/* { dg-do compile } */
+/* { dg-options "-Wbidi-chars=ucn,any" } */
+
+const char *
+fn ()
+{
+  const char *aText = "\u202D" "abc";
+/* { dg-warning "U\\+202D" "" { target *-*-* } .-1 } */
+  return aText;
+}
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-4.c b/gcc/testsuite/c-c++-common/Wbidi-chars-4.c
index 639e5c62e88..d2f0739dae0 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-4.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-4.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=any -Wno-multichar -Wno-overflow" } */
+/* { dg-options "-Wbidi-chars=any,ucn -Wno-multichar -Wno-overflow" } */
 /* Test all bidi chars in various contexts (identifiers, comments,
    string literals, character constants), both UCN and UTF-8.  The bidi
    chars here are properly terminated, except for the character constants.  */
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-5.c b/gcc/testsuite/c-c++-common/Wbidi-chars-5.c
index 68cb053144b..ad49498fe23 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-5.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-5.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired -Wno-multichar -Wno-overflow" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn -Wno-multichar -Wno-overflow" } */
 /* Test all bidi chars in various contexts (identifiers, comments,
    string literals, character constants), both UCN and UTF-8.  The bidi
    chars here are properly terminated, except for the character constants.  */
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-6.c b/gcc/testsuite/c-c++-common/Wbidi-chars-6.c
index 0ce6fff2dee..8c1c1b2a270 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-6.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-6.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=ucn,unpaired" } */
 /* Test nesting of bidi chars in various contexts.  */
 
 /* Terminated by the wrong char:  */
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-7.c b/gcc/testsuite/c-c++-common/Wbidi-chars-7.c
index d012d420ec0..3270952a09a 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-7.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-7.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=any" } */
+/* { dg-options "-Wbidi-chars=any,ucn" } */
 /* Test we ignore UCNs in comments.  */
 
 // a b c \u202a 1 2 3
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-8.c b/gcc/testsuite/c-c++-common/Wbidi-chars-8.c
index 4f54c5092ec..3983168c9f1 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-8.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-8.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=any" } */
+/* { dg-options "-Wbidi-chars=any,ucn" } */
 /* Test \u vs \U.  */
 
 int a_\u202A;
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-9.c b/gcc/testsuite/c-c++-common/Wbidi-chars-9.c
index e2af1b1ca97..0ddb0d93108 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-9.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-9.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn" } */
 /* Test that we properly separate bidi contexts (comment/identifier/character
    constant/string literal).  */
 
diff --git a/gcc/testsuite/c-c++-common/Wbidi-chars-ranges.c b/gcc/testsuite/c-c++-common/Wbidi-chars-ranges.c
index 298750a2a64..0c71f306dbc 100644
--- a/gcc/testsuite/c-c++-common/Wbidi-chars-ranges.c
+++ b/gcc/testsuite/c-c++-common/Wbidi-chars-ranges.c
@@ -1,6 +1,6 @@ 
 /* PR preprocessor/103026 */
 /* { dg-do compile } */
-/* { dg-options "-Wbidi-chars=unpaired -fdiagnostics-show-caret" } */
+/* { dg-options "-Wbidi-chars=unpaired,ucn -fdiagnostics-show-caret" } */
 /* Verify that we escape and underline pertinent bidirectional
    control characters when quoting the source.  */
 
diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
index 940c79f98c1..9661b7151b1 100644
--- a/libcpp/include/cpplib.h
+++ b/libcpp/include/cpplib.h
@@ -327,7 +327,10 @@  enum cpp_bidirectional_level {
   /* Only detect unpaired uses of bidirectional control characters.  */
   bidirectional_unpaired,
   /* Detect any use of bidirectional control characters.  */
-  bidirectional_any
+  bidirectional_any,
+  /* Like above, but also warn about UCNs.  */
+  bidirectional_unpaired_ucn,
+  bidirectional_any_ucn
 };
 
 /* This structure is nested inside struct cpp_reader, and
diff --git a/libcpp/lex.cc b/libcpp/lex.cc
index 4d736576cc1..348ec553b7a 100644
--- a/libcpp/lex.cc
+++ b/libcpp/lex.cc
@@ -1560,8 +1560,11 @@  class unpaired_bidi_rich_location : public rich_location
 static void
 maybe_warn_bidi_on_close (cpp_reader *pfile, const uchar *p)
 {
-  if (CPP_OPTION (pfile, cpp_warn_bidirectional) == bidirectional_unpaired
-      && bidi::vec.count () > 0)
+  const auto warn_bidi = CPP_OPTION (pfile, cpp_warn_bidirectional);
+  if (bidi::vec.count () > 0
+      && (warn_bidi == bidirectional_unpaired_ucn
+	  || (warn_bidi == bidirectional_unpaired
+	      && !bidi::current_ctx_ucn_p ())))
     {
       const location_t loc
 	= linemap_position_for_column (pfile->line_table,
@@ -1605,10 +1608,10 @@  maybe_warn_bidi_on_char (cpp_reader *pfile, bidi::kind kind,
       /* It seems excessive to warn about a PDI/PDF that is closing
 	 an opened context because we've already warned about the
 	 opening character.  Except warn when we have a UCN x UTF-8
-	 mismatch.  */
+	 mismatch, if UCN checking is enabled.  */
       if (kind == bidi::current_ctx ())
 	{
-	  if (warn_bidi == bidirectional_unpaired
+	  if (warn_bidi == bidirectional_unpaired_ucn
 	      && bidi::current_ctx_ucn_p () != ucn_p)
 	    {
 	      rich_loc.add_range (bidi::current_ctx_loc ());
@@ -1617,7 +1620,8 @@  maybe_warn_bidi_on_char (cpp_reader *pfile, bidi::kind kind,
 			      "a context by \"%s\"", bidi::to_str (kind));
 	    }
 	}
-      else if (warn_bidi == bidirectional_any)
+      else if (warn_bidi == bidirectional_any_ucn
+	       || (warn_bidi == bidirectional_any && !ucn_p))
 	{
 	  if (kind == bidi::kind::PDF || kind == bidi::kind::PDI)
 	    cpp_warning_at (pfile, CPP_W_BIDIRECTIONAL, &rich_loc,