PE/COFF: auto-promote to bigobj target when section count overflows

  Regular PE/COFF uses 16-bit section numbers in the symbol table,
limiting objects to at most 65535 sections.  When this limit is
exceeded, the assembler silently truncates section numbers, producing
corrupt objects that cause confusing linker errors like:

  undefined reference to `vtable for ...'

Fix this by automatically switching the xvec to the matching bigobj
target before any headers or symbols are finalized, so the object is
written out using the extended COFF format which supports 32-bit
section numbers.

bfd/
	* coffcode.h (coff_compute_section_file_positions): Auto-promote
	to bigobj target when section count exceeds 16-bit limit.

gas/
	* testsuite/gas/pe/big-obj-auto.d: New test.
	* testsuite/gas/pe/big-obj-auto.s: New test.
	* testsuite/gas/pe/pe.exp: Run big-obj-auto.

Signed-off-by: Oleg Tolmatcev <oleg.tolmatcev@gmail.com>
---
 bfd/coffcode.h                      | 14 ++++++++++++++
 gas/testsuite/gas/pe/big-obj-auto.d |  9 +++++++++
 gas/testsuite/gas/pe/big-obj-auto.s | 16 ++++++++++++++++
 gas/testsuite/gas/pe/pe.exp         |  1 +
 4 files changed, 40 insertions(+)
 create mode 100644 gas/testsuite/gas/pe/big-obj-auto.d
 create mode 100644 gas/testsuite/gas/pe/big-obj-auto.s
  

On 21.06.2026 19:49, Oleg Tolmatcev wrote:
> Regular PE/COFF uses 16-bit section numbers in the symbol table,
> limiting objects to at most 65535 sections.  When this limit is
> exceeded, the assembler silently truncates section numbers, producing
> corrupt objects that cause confusing linker errors like:
> 
>   undefined reference to `vtable for ...'
> 
> Fix this by automatically switching the xvec to the matching bigobj
> target before any headers or symbols are finalized, so the object is
> written out using the extended COFF format which supports 32-bit
> section numbers.

It has never really become clear to me why this wouldn't have been the
default behavior from the very beginning. Perhaps with a way to opt out
(rather than one to opt in).

> --- a/bfd/coffcode.h
> +++ b/bfd/coffcode.h
> @@ -360,6 +360,10 @@ CODE_FRAGMENT
>  #include "coffswap.h"
>  #endif
>  
> +#ifdef COFF_WITH_PE_BIGOBJ
> +extern const bfd_target TARGET_SYM_BIG;
> +#endif
> +
>  #define STRING_SIZE_SIZE 4
>  
>  #define DOT_DEBUG	".debug"
> @@ -3154,6 +3158,16 @@ coff_compute_section_file_positions (bfd * abfd)
>  
>    if (target_index >= bfd_coff_max_nscns (abfd))
>      {
> +#ifdef COFF_WITH_PE_BIGOBJ
> +      if (abfd->xvec != &TARGET_SYM_BIG)
> +	{
> +	  /* Regular PE/COFF uses 16-bit section numbers in symbols.
> +	     Promote oversized objects to the matching bigobj target
> +	     before any headers or symbols are finalized.  */
> +	  abfd->xvec = &TARGET_SYM_BIG;
> +	  return coff_compute_section_file_positions (abfd);
> +	}
> +#endif
>        bfd_set_error (bfd_error_file_too_big);
>        _bfd_error_handler
>  	/* xgettext:c-format */

In the description you mention silent generation of a corrupt output. Yet then
you place your addition next to the emission of an error. If this error path
wasn't taken in the case of interest, your new addition also would take any
effect. What am I missing?

> --- /dev/null
> +++ b/gas/testsuite/gas/pe/big-obj-auto.d
> @@ -0,0 +1,9 @@
> +#objdump: -h
> +#name: PE big obj auto-promotion
> +
> +.*: *file format pe-bigobj-.*
> +
> +Sections:
> +#...
> +5000. \.data\$a49999  .*

5000. still fits in 16 bits. How is this sufficient a test for the new behavior?
I expect you want ...

> --- /dev/null
> +++ b/gas/testsuite/gas/pe/big-obj-auto.s
> @@ -0,0 +1,16 @@
> +	.file "big-obj-auto.s"
> +
> +	.irp n,0,1,2,3,4

	.irp n,0,1,2,3,4,5,6,7

here.

> +	.irp m,0,1,2,3,4,5,6,7,8,9
> +	.irp c,0,1,2,3,4,5,6,7,8,9
> +	.irp d,0,1,2,3,4,5,6,7,8,9
> +	.irp u,0,1,2,3,4,5,6,7,8,9
> +	.globl a\n\m\c\d\u
> +	.section .data$a\n\m\c\d\u,"w"
> +a\n\m\c\d\u :
> +	.byte 1
> +	.endr
> +	.endr
> +	.endr
> +	.endr
> +	.endr
> --- a/gas/testsuite/gas/pe/pe.exp
> +++ b/gas/testsuite/gas/pe/pe.exp
> @@ -70,4 +70,5 @@ if {[istarget "aarch64-*-pe*"] || [istarget "aarch64-*-mingw*"]} {
>  
>  if ([istarget "*-*-mingw*"]) then {
>  	run_dump_test "big-obj"
> +	run_dump_test "big-obj-auto"
>  }

Why would the test want limiting to MinGW?

Jan
  

Am Mo., 22. Juni 2026 um 09:46 Uhr schrieb Jan Beulich <jbeulich@suse.com>:
>
> On 21.06.2026 19:49, Oleg Tolmatcev wrote:
> > Regular PE/COFF uses 16-bit section numbers in the symbol table,
> > limiting objects to at most 65535 sections.  When this limit is
> > exceeded, the assembler silently truncates section numbers, producing
> > corrupt objects that cause confusing linker errors like:
> >
> >   undefined reference to `vtable for ...'
> >
> > Fix this by automatically switching the xvec to the matching bigobj
> > target before any headers or symbols are finalized, so the object is
> > written out using the extended COFF format which supports 32-bit
> > section numbers.
>
> It has never really become clear to me why this wouldn't have been the
> default behavior from the very beginning. Perhaps with a way to opt out
> (rather than one to opt in).
>
> > --- a/bfd/coffcode.h
> > +++ b/bfd/coffcode.h
> > @@ -360,6 +360,10 @@ CODE_FRAGMENT
> >  #include "coffswap.h"
> >  #endif
> >
> > +#ifdef COFF_WITH_PE_BIGOBJ
> > +extern const bfd_target TARGET_SYM_BIG;
> > +#endif
> > +
> >  #define STRING_SIZE_SIZE 4
> >
> >  #define DOT_DEBUG    ".debug"
> > @@ -3154,6 +3158,16 @@ coff_compute_section_file_positions (bfd * abfd)
> >
> >    if (target_index >= bfd_coff_max_nscns (abfd))
> >      {
> > +#ifdef COFF_WITH_PE_BIGOBJ
> > +      if (abfd->xvec != &TARGET_SYM_BIG)
> > +     {
> > +       /* Regular PE/COFF uses 16-bit section numbers in symbols.
> > +          Promote oversized objects to the matching bigobj target
> > +          before any headers or symbols are finalized.  */
> > +       abfd->xvec = &TARGET_SYM_BIG;
> > +       return coff_compute_section_file_positions (abfd);
> > +     }
> > +#endif
> >        bfd_set_error (bfd_error_file_too_big);
> >        _bfd_error_handler
> >       /* xgettext:c-format */
>
> In the description you mention silent generation of a corrupt output. Yet then
> you place your addition next to the emission of an error. If this error path
> wasn't taken in the case of interest, your new addition also would take any
> effect. What am I missing?

I looked into it and discovered that the silent corruption was not caused by
upstream code. I am using MSYS2 UCRT64 and MSYS2 applies a patch
"0010-bfd-Increase-_bfd_coff_max_nscns-to-65279.patch" which raises
the maximum section number from 32768 to 65279. This disables
the check that produces the error message "too many sections" and
leads to silent miscompilation because the code still treats the section
numbers as signed integers.

I didn't know that and that's why the commit message is wrong. The
fix itself is correct though. I guess I'll need to send a v2 of the patch
with a new message.

> > --- /dev/null
> > +++ b/gas/testsuite/gas/pe/big-obj-auto.d
> > @@ -0,0 +1,9 @@
> > +#objdump: -h
> > +#name: PE big obj auto-promotion
> > +
> > +.*: *file format pe-bigobj-.*
> > +
> > +Sections:
> > +#...
> > +5000. \.data\$a49999  .*
>
> 5000. still fits in 16 bits. How is this sufficient a test for the new behavior?
> I expect you want ...

I can fix it in v2 of the patch.

> > --- /dev/null
> > +++ b/gas/testsuite/gas/pe/big-obj-auto.s
> > @@ -0,0 +1,16 @@
> > +     .file "big-obj-auto.s"
> > +
> > +     .irp n,0,1,2,3,4
>
>         .irp n,0,1,2,3,4,5,6,7
>
> here.
>
> > +     .irp m,0,1,2,3,4,5,6,7,8,9
> > +     .irp c,0,1,2,3,4,5,6,7,8,9
> > +     .irp d,0,1,2,3,4,5,6,7,8,9
> > +     .irp u,0,1,2,3,4,5,6,7,8,9
> > +     .globl a\n\m\c\d\u
> > +     .section .data$a\n\m\c\d\u,"w"
> > +a\n\m\c\d\u :
> > +     .byte 1
> > +     .endr
> > +     .endr
> > +     .endr
> > +     .endr
> > +     .endr
> > --- a/gas/testsuite/gas/pe/pe.exp
> > +++ b/gas/testsuite/gas/pe/pe.exp
> > @@ -70,4 +70,5 @@ if {[istarget "aarch64-*-pe*"] || [istarget "aarch64-*-mingw*"]} {
> >
> >  if ([istarget "*-*-mingw*"]) then {
> >       run_dump_test "big-obj"
> > +     run_dump_test "big-obj-auto"
> >  }
>
> Why would the test want limiting to MinGW?

I don't know because I didn't write that code.

> Jan

Oleg
  

On 22.06.2026 21:26, Oleg Tolmatcev wrote:
> Am Mo., 22. Juni 2026 um 09:46 Uhr schrieb Jan Beulich <jbeulich@suse.com>:
>>> --- a/gas/testsuite/gas/pe/pe.exp
>>> +++ b/gas/testsuite/gas/pe/pe.exp
>>> @@ -70,4 +70,5 @@ if {[istarget "aarch64-*-pe*"] || [istarget "aarch64-*-mingw*"]} {
>>>
>>>  if ([istarget "*-*-mingw*"]) then {
>>>       run_dump_test "big-obj"
>>> +     run_dump_test "big-obj-auto"
>>>  }
>>
>> Why would the test want limiting to MinGW?
> 
> I don't know because I didn't write that code.

You didn't write the original code, but it was you who decided to insert the
new test's running inside the conditional, rather than outside (or in a new,
less restrictive one). Especially since the new "default to big-obj if needed"
isn't (directly) target dependent, the new test also should cover all targets
which are affected by the change.

Of course there's the orthogonal issue of the pre-existing conditional wanting
relaxing. That's not something I'm demanding you to do at the same time (yet
of course I wouldn't mind if you did). The three bfd/pe-*.c which #define
COFF_WITH_PE_BIGOBJ do so entirely unconditionally, afaics. Hence that's the
pattern we want to also use for engaging the pre-existing and the new test.

Jan