fortran: Fix treatment of UTF-8 byte order marks in source files [PR128343]

Message ID 559d0b60-7a53-4c7b-9f66-a447b4cb6ab2@gmx.de
State New
Headers
Series fortran: Fix treatment of UTF-8 byte order marks in source files [PR128343] |

Checks

Context Check Description
linaro-tcwg-bot/tcwg_gcc_build--master-aarch64 success Build passed
linaro-tcwg-bot/tcwg_gcc_build--master-arm success Build passed
linaro-tcwg-bot/tcwg_simplebootstrap_build--master-aarch64-bootstrap success Build passed
linaro-tcwg-bot/tcwg_simplebootstrap_build--master-arm-bootstrap success Build passed

Commit Message

Harald Anlauf Nov. 3, 2026, 10:39 p.m. UTC
  Dear all,

this patch seems to be a very special challenge:

The testcase has an UTF-8 byte order mark, which I do not see in the
patch unless I use "od".  If you don't see it, then please see the
attachment in the PR for reference.

We used a too short line buffer when reading the testcase, leading
to valgrind invalid read/writes.  The solution here is to extend
the line buffer, taking care of line length limits.  I do hope I
got it right.

Regtested on x86_64-pc-linux-gnu.  OK for mainline?

Thanks,
Harald
  

Comments

Jerry D Nov. 4, 2026, 12:08 a.m. UTC | #1
On 11/3/26 2:39 PM, Harald Anlauf wrote:
> Dear all,
> 
> this patch seems to be a very special challenge:
> 
> The testcase has an UTF-8 byte order mark, which I do not see in the
> patch unless I use "od".  If you don't see it, then please see the
> attachment in the PR for reference.
> 
> We used a too short line buffer when reading the testcase, leading
> to valgrind invalid read/writes.  The solution here is to extend
> the line buffer, taking care of line length limits.  I do hope I
> got it right.
> 
> Regtested on x86_64-pc-linux-gnu.  OK for mainline?
> 
> Thanks,
> Harald

Yes, OK Harald.

I did a read of the standard and this is acceptable.

Jerry
  
Harald Anlauf Nov. 4, 2026, 10:54 p.m. UTC | #2
Hi Jerry, all,

Am 04.11.26 um 1:08 AM schrieb Jerry D:
> On 11/3/26 2:39 PM, Harald Anlauf wrote:
>> Dear all,
>>
>> this patch seems to be a very special challenge:
>>
>> The testcase has an UTF-8 byte order mark, which I do not see in the
>> patch unless I use "od".  If you don't see it, then please see the
>> attachment in the PR for reference.
>>
>> We used a too short line buffer when reading the testcase, leading
>> to valgrind invalid read/writes.  The solution here is to extend
>> the line buffer, taking care of line length limits.  I do hope I
>> got it right.
>>
>> Regtested on x86_64-pc-linux-gnu.  OK for mainline?
>>
>> Thanks,
>> Harald
> 
> Yes, OK Harald.
> 
> I did a read of the standard and this is acceptable.

thanks, this is what I hoped for.

Since this thread is growing, I would like to remind all that
my patch does not change the behavior of gfortran other than
to prevent a buffer overrun.

The current behavior of gfortran (scanner.cc:load_file) was implemented
by FX in 2007/2008.  His PR mentions
   https://en.wikipedia.org/wiki/Byte_order_mark

commit caef7872f0dc4a62dfc449785974eaa179b0a449
Author: Francois-Xavier Coudert <fxcoudert@gcc.gnu.org>
Date:   Sun Apr 29 10:45:57 2007 +0000

     re PR fortran/31645 (Error on reading Byte Order Mark)

             PR fortran/31645

             * scanner.c (load_file): Discard the byte order mark if one is
             found on the first non-preprocessor line of a file.


       /* If this is the first line of the file, it can contain a byte
	 order mark (BOM), which we will ignore:
	   FF FE is UTF-16 little endian,
	   FE FF is UTF-16 big endian,
	   EF BB BF is UTF-8.  */
       if (first_line
	  && ((line_len >= 2 && line[0] == (unsigned char) '\xFF'
			     && line[1] == (unsigned char) '\xFE')
	      || (line_len >= 2 && line[0] == (unsigned char) '\xFE'
			        && line[1] == (unsigned char) '\xFF')
	      || (line_len >= 3 && line[0] == (unsigned char) '\xEF'
				&& line[1] == (unsigned char) '\xBB'
				&& line[2] == (unsigned char) '\xBF')))
	{
...

This behavior is tested by a couple of testcases, so I am not too happy
if there is the requirement to change the default.

Note the BOM will be removed only when it is at the beginning of the
first line, and only then the column count should be adjusted.

Then I did not take care of UTF-16 or UTF-32 in particular.  I just
ran the testcases under valgrind but did not touch anything here.

A slightly modified version of the testcase (see attached) still gives:

% gfc-17 pr128343-v2.f -w
pr128343-v2.f:24:2:

    24 | ! <- here's an unsolicited BOM { dg-error "Invalid character 
0xBB" }
       |  1
Error: Non-numeric character in statement label at (1)
pr128343-v2.f:24:3:

    24 | ! <- here's an unsolicited BOM { dg-error "Invalid character 
0xBB" }
       |
Error: Invalid character 0xBB at (1)

I get the same as with 16 modulo the (previous) risk of memory
corruption.

If there are strong opinions, I can add a warning under a non-default
-W option if a BOM is found so that existing testcases remain happy.

Harald


> Jerry
>
  
FX Coudert Nov. 5, 2026, 8:32 a.m. UTC | #3
Hi,

> The current behavior of gfortran (scanner.cc:load_file) was implemented
> by FX in 2007/2008.  His PR mentions
>  https://en.wikipedia.org/wiki/Byte_order_mark

The presence of a BOM as first character of a file is different from the use of U+FEFF ZERO WIDTH NO-BREAK SPACE as a character in the rest of the file.
Since it’s been handled in a particular way for 20 years with no complaints, and in line with other compilers, I don’t see why it would be important to change the current behavior (other than fix the memory issue).

FX
  
Harald Anlauf Nov. 5, 2026, 7:56 p.m. UTC | #4
Am 05.11.26 um 9:32 AM schrieb FX Coudert:
> Hi,
> 
>> The current behavior of gfortran (scanner.cc:load_file) was implemented
>> by FX in 2007/2008.  His PR mentions
>>   https://en.wikipedia.org/wiki/Byte_order_mark
> 
> The presence of a BOM as first character of a file is different from the use of U+FEFF ZERO WIDTH NO-BREAK SPACE as a character in the rest of the file.
> Since it’s been handled in a particular way for 20 years with no complaints, and in line with other compilers, I don’t see why it would be important to change the current behavior (other than fix the memory issue).
> 
> FX

Thanks, FX.

Before pushing, because the existing testcases for BOM are free form,
I decided to rename the new testcase (fixed form!) to bom_UTF-8-2.f .

Pushed as r17-5986-gbf136cb5649057 to mainline.

Harald
  

Patch

From 8f7db1002c2a98a9195c56313b4501e73c89524b Mon Sep 17 00:00:00 2001
From: Harald Anlauf <anlauf@gmx.de>
Date: Tue, 3 Nov 2026 23:19:53 +0100
Subject: [PATCH] fortran: Fix treatment of UTF-8 byte order marks in source
 files [PR128343]

When a gfortran source file contains a BOM, we ignore it, but only on the
first line of a file.  In that case the temporary line buffer needs to be
extended to avoid valgrind errors.

	PR fortran/128343

gcc/fortran/ChangeLog:

	* scanner.cc (load_line): Check for optional UTF-8 BOM at beginning
	of line.  Adjust temporary line buffer length and line truncation.
	(load_file): Use it when reading the very first line of a source
	file.

gcc/testsuite/ChangeLog:

	* gfortran.dg/pr128343.f: New test.
---
 gcc/fortran/scanner.cc               | 33 ++++++++++++++++++++++++----
 gcc/testsuite/gfortran.dg/pr128343.f | 22 +++++++++++++++++++
 2 files changed, 51 insertions(+), 4 deletions(-)
 create mode 100644 gcc/testsuite/gfortran.dg/pr128343.f

diff --git a/gcc/fortran/scanner.cc b/gcc/fortran/scanner.cc
index 9b77e559065..85b25f77da5 100644
--- a/gcc/fortran/scanner.cc
+++ b/gcc/fortran/scanner.cc
@@ -1801,6 +1801,9 @@  gfc_gobble_whitespace (void)
    the first character of the line, which has already been read by the
    caller.  This avoids the use of ungetc().
 
+   If allow_bom is true, increase length of line buffer to allow for UTF-8 BOM
+   (byte order mark) at the beginning of the line.
+
    load_line returns whether the line was truncated.
 
    NOTE: The error machinery isn't available at this point, so we can't
@@ -1808,7 +1811,8 @@  gfc_gobble_whitespace (void)
 	 parts of gfortran.  */
 
 static bool
-load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char)
+load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char,
+	   bool allow_bom = false)
 {
   int c, maxlen, i, preprocessor_flag, buflen = *pbuflen;
   int quoted = ' ', comment_ix = -1;
@@ -1819,6 +1823,7 @@  load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char)
   bool seen_ampersand = false;
   bool found_tab = false;
   bool warned_tabs = false;
+  bool seen_utf8_bom = false;
   gfc_char_t *buffer;
 
   /* Determine the maximum allowed line length.  */
@@ -1841,6 +1846,10 @@  load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char)
       else
 	buflen = 132;
 
+      /* Increase line buffer length for BOM.  */
+      if (allow_bom)
+	buflen += 3;
+
       *pbuf = gfc_get_wide_string (buflen + 1);
     }
 
@@ -1977,6 +1986,18 @@  load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char)
       *buffer++ = c;
       i++;
 
+      /* Check for possible UTF-8 BOM.  */
+      if (allow_bom
+	  && i == 3
+	  && buffer[-3] == (unsigned char) '\xEF'
+	  && buffer[-2] == (unsigned char) '\xBB'
+	  && buffer[-1] == (unsigned char) '\xBF')
+	seen_utf8_bom = true;
+
+      /* Check for sentinel '#' after BOM.  */
+      if (i == 4 && seen_utf8_bom)
+	preprocessor_flag = (c == '#');
+
       if (maxlen == 0 || preprocessor_flag)
 	{
 	  if (i >= buflen)
@@ -1988,7 +2009,7 @@  load_line (FILE *input, gfc_char_t **pbuf, int *pbuflen, const int *first_char)
 	      buffer = (*pbuf) + i;
 	    }
 	}
-      else if (i >= maxlen)
+      else if (i >= maxlen + (seen_utf8_bom ? 3 : 0))
 	{
 	  bool trunc_warn = true;
 
@@ -2035,7 +2056,7 @@  next_char:
       && !preprocessor_flag
       && c != EOF)
     {
-      while (i++ < maxlen)
+      while (i++ < maxlen + (seen_utf8_bom ? 3 : 0))
 	*buffer++ = ' ';
     }
 
@@ -2541,6 +2562,7 @@  load_file (const char *realfilename, const char *displayedname, bool initial)
   FILE *input;
   int len, line_len;
   bool first_line;
+  bool allow_bom;
   struct stat st;
   int stat_result;
   const char *filename;
@@ -2632,11 +2654,14 @@  load_file (const char *realfilename, const char *displayedname, bool initial)
 	}
     }
 
+  /* Allow BOM on very first line of file.  */
+  allow_bom = true;
   for (;;)
     {
-      int trunc = load_line (input, &line, &line_len, NULL);
+      int trunc = load_line (input, &line, &line_len, NULL, allow_bom);
       int inc_line;
 
+      allow_bom = false;
       len = gfc_wide_strlen (line);
       if (feof (input) && len == 0)
 	break;
diff --git a/gcc/testsuite/gfortran.dg/pr128343.f b/gcc/testsuite/gfortran.dg/pr128343.f
new file mode 100644
index 00000000000..e7767eb0897
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/pr128343.f
@@ -0,0 +1,22 @@ 
+#define a(i) b(i)
+!
+! PR fortran/128343 - UTF-8 byte order mark and internal line buffer
+!
+! { dg-additional-options "-w" }
+!
+! Note: this file has the following characters at the beginning:
+!
+! 0000000 357 273 277 043 144 145 146 151 156 145 040 141 050 151 051 040
+!         357 273 277   #   d   e   f   i   n   e       a   (   i   )    
+!
+! which lead to invalid write/read errors under valgrind.
+!
+! Derived from 1066_0007.f from the Fujitsu test suite on Github.
+
+#if 0
+#else ! a
+      ! a
+      write(10,*) a
+      print *,'pass'
+#endif ! a a
+      end
-- 
2.51.0