[v1] tree-ssa-math-opts: separate FMA-deferring trigger from reassoc reorder
Checks
Commit Message
param_avoid_fma_max_bits gates two independent transforms: the
widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
loop-carried multiply-add as fmul + fadd, and the reassoc
loop-carried-FMA reorder added in r14-5779-g746344dd538
(PR tree-optimization/110279), which parallelises 3+ operand chains.
Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
type they switch on at the same threshold and a target cannot keep one
while dropping the other.
This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
family): a 2-operand reduction such as an sgemm inner K-loop is left as
fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
Add a generic -fwidening-mul-defer-fma (default on) that gates only the
widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
target and no default behaviour changes.
gcc/ChangeLog:
* common.opt (fwidening-mul-defer-fma): New option.
* tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
Gate fma_deferring_state's enable predicate on
flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
* config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
Update the comment for the -fwidening-mul-defer-fma effect.
* config/aarch64/aarch64.cc (aarch64_override_options_internal):
Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
flag_widening_mul_defer_fma.
* doc/invoke.texi (-fwidening-mul-defer-fma): Document.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
* gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
* gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
---
gcc/common.opt | 4 ++++
gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
gcc/config/aarch64/aarch64.cc | 12 ++++++++---
gcc/doc/invoke.texi | 17 ++++++++++++++-
.../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
.../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
.../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
gcc/tree-ssa-math-opts.cc | 3 ++-
8 files changed, 95 insertions(+), 5 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
Comments
On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
>
> param_avoid_fma_max_bits gates two independent transforms: the
> widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
> loop-carried multiply-add as fmul + fadd, and the reassoc
> loop-carried-FMA reorder added in r14-5779-g746344dd538
> (PR tree-optimization/110279), which parallelises 3+ operand chains.
> Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
> type they switch on at the same threshold and a target cannot keep one
> while dropping the other.
>
> This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
> family): a 2-operand reduction such as an sgemm inner K-loop is left as
> fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
> setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
>
> Add a generic -fwidening-mul-defer-fma (default on) that gates only the
> widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
> alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
> target and no default behaviour changes.
Note this is just a review on the documentation and nothing else.
>
> gcc/ChangeLog:
>
> * common.opt (fwidening-mul-defer-fma): New option.
> * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
> Gate fma_deferring_state's enable predicate on
> flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
> * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
> Update the comment for the -fwidening-mul-defer-fma effect.
> * config/aarch64/aarch64.cc (aarch64_override_options_internal):
> Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
> flag_widening_mul_defer_fma.
> * doc/invoke.texi (-fwidening-mul-defer-fma): Document.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
> * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
> * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
>
> Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
> ---
>
> gcc/common.opt | 4 ++++
> gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
> gcc/config/aarch64/aarch64.cc | 12 ++++++++---
> gcc/doc/invoke.texi | 17 ++++++++++++++-
> .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
> .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
> .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
> gcc/tree-ssa-math-opts.cc | 3 ++-
> 8 files changed, 95 insertions(+), 5 deletions(-)
> create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
>
> diff --git a/gcc/common.opt b/gcc/common.opt
> index 218dddf5dfe9..9be7dd6b2315 100644
> --- a/gcc/common.opt
> +++ b/gcc/common.opt
> @@ -3639,6 +3639,10 @@ fweb
> Common Var(flag_web) Optimization EnabledBy(funroll-loops)
> Construct webs and split unrelated uses of single variable.
>
> +fwidening-mul-defer-fma
> +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
> +Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
Don't mention --param in user documentation directly.
> +
> ftree-builtin-call-dce
> Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
> Enable conditional dead code elimination for builtin calls.
> diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def b/gcc/config/aarch64/aarch64-tuning-flags.def
> index 058dadecccaa..5b087bfc9f03 100644
> --- a/gcc/config/aarch64/aarch64-tuning-flags.def
> +++ b/gcc/config/aarch64/aarch64-tuning-flags.def
> @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
>
> AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
>
> +/* For cores whose pipeline disfavours loop-carried serial FMAs: set
> + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
> + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
> + pass's FMA deferring. */
> AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
>
> AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index 78f1eae8336c..fdb5bbbdfbce 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
> && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
> opts->x_flag_prefetch_loop_arrays = 1;
>
> - /* Avoid loop-dependant FMA chains. */
> + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
> + keeps using --param=avoid-fma-max-bits; the widening-mul-side
> + deferring is gated separately by -fwidening-mul-defer-fma, so we
> + suppress only the deferring on these cores while leaving the reassoc
> + reorder active. */
> if (aarch64_tune_params.extra_tuning_flags
> & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
> - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
> - 512);
> + {
> + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
> + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
> + }
>
> /* Consider fully pipelined FMA in reassociation. */
> if (aarch64_tune_params.extra_tuning_flags
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index 208625cafc55..f39902f77e9b 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
> -funconstrained-commons -funit-at-a-time -funroll-all-loops
> -funroll-loops -funsafe-math-optimizations -funswitch-loops
> -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
> --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
> +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
> +-fuse-linker-plugin -fzero-call-used-regs
> -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
>
> @item Program Instrumentation Options
> @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
>
> Enabled by default with @option{-funroll-loops}.
>
> +@opindex fwidening-mul-defer-fma
> +@opindex fno-widening-mul-defer-fma
> +@item -fwidening-mul-defer-fma
> +When FMA avoidance is requested for loop-carried reductions (that is, when
> +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
> +refuses to contract a multiply-add whose result feeds a loop-header PHI,
> +leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
> +to suppress this deferring and contract such reductions into an FMA, while
> +still leaving @option{--param=avoid-fma-max-bits} in effect for the
> +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
> +This is useful on targets whose pipeline disfavours the extra floating-point
> +operation more than the loop-carried FMA latency. The option is enabled by
> +default.
Here either.
> +
> @opindex fwhole-program
> @opindex fno-whole-program
> @item -fwhole-program
> diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> new file mode 100644
> index 000000000000..b542274eb763
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> @@ -0,0 +1,19 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
> +
> +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
> + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
> + reduction is contracted to fmadd rather than deferred to fmul + fadd.
> + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
> +
> +double
> +dot (const double *a, const double *b, int n)
> +{
> + double s = 0.0;
> + for (int i = 0; i < n; i++)
> + s += a[i] * b[i];
> + return s;
> +}
> +
> +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> new file mode 100644
> index 000000000000..764dd2457435
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> @@ -0,0 +1,21 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
> +
> +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
> + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
> + whose result feeds the loop-header phi: the reduction stays as a
> + separate fmul + fadd. This is the behaviour the new flag decouples
> + from the reassoc reorder (also gated by avoid-fma-max-bits). */
> +
> +double
> +dot (const double *a, const double *b, int n)
> +{
> + double s = 0.0;
> + for (int i = 0; i < n; i++)
> + s += a[i] * b[i];
> + return s;
> +}
> +
> +/* { dg-final { scan-assembler {\tfmul\t} } } */
> +/* { dg-final { scan-assembler {\tfadd\t} } } */
> +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
> diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> new file mode 100644
> index 000000000000..5e041afc18fc
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
> +
> +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
> + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
> + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
> + so the loop-carried multiply-add is contracted to a single fmadd. This
> + proves the new flag decouples the deferring from avoid-fma-max-bits. */
> +
> +double
> +dot (const double *a, const double *b, int n)
> +{
> + double s = 0.0;
> + for (int i = 0; i < n; i++)
> + s += a[i] * b[i];
> + return s;
> +}
> +
> +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
> index f0ede668d95e..a00146922713 100644
> --- a/gcc/tree-ssa-math-opts.cc
> +++ b/gcc/tree-ssa-math-opts.cc
> @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
> {
> gimple_stmt_iterator gsi;
>
> - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
> + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
> + && flag_widening_mul_defer_fma);
>
> for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
> psi = psi_next)
> --
> 2.34.1
>
> base-commit: 3b443e764144526748d2b51d017d745f1956377e
> branch: ptomsich/475-widening-mul-defer-fma
On Tue, 7 Jul 2026 at 23:40, Andrea Pinski
<andrew.pinski@oss.qualcomm.com> wrote:
>
> On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> >
> > param_avoid_fma_max_bits gates two independent transforms: the
> > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
> > loop-carried multiply-add as fmul + fadd, and the reassoc
> > loop-carried-FMA reorder added in r14-5779-g746344dd538
> > (PR tree-optimization/110279), which parallelises 3+ operand chains.
> > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
> > type they switch on at the same threshold and a target cannot keep one
> > while dropping the other.
> >
> > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
> > family): a 2-operand reduction such as an sgemm inner K-loop is left as
> > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
> > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
> >
> > Add a generic -fwidening-mul-defer-fma (default on) that gates only the
> > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
> > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
> > target and no default behaviour changes.
>
> Note this is just a review on the documentation and nothing else.
>
> >
> > gcc/ChangeLog:
> >
> > * common.opt (fwidening-mul-defer-fma): New option.
> > * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
> > Gate fma_deferring_state's enable predicate on
> > flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
> > * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
> > Update the comment for the -fwidening-mul-defer-fma effect.
> > * config/aarch64/aarch64.cc (aarch64_override_options_internal):
> > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
> > flag_widening_mul_defer_fma.
> > * doc/invoke.texi (-fwidening-mul-defer-fma): Document.
> >
> > gcc/testsuite/ChangeLog:
> >
> > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
> > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
> > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
> >
> > Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
> > ---
> >
> > gcc/common.opt | 4 ++++
> > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
> > gcc/config/aarch64/aarch64.cc | 12 ++++++++---
> > gcc/doc/invoke.texi | 17 ++++++++++++++-
> > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
> > .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
> > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
> > gcc/tree-ssa-math-opts.cc | 3 ++-
> > 8 files changed, 95 insertions(+), 5 deletions(-)
> > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> >
> > diff --git a/gcc/common.opt b/gcc/common.opt
> > index 218dddf5dfe9..9be7dd6b2315 100644
> > --- a/gcc/common.opt
> > +++ b/gcc/common.opt
> > @@ -3639,6 +3639,10 @@ fweb
> > Common Var(flag_web) Optimization EnabledBy(funroll-loops)
> > Construct webs and split unrelated uses of single variable.
> >
> > +fwidening-mul-defer-fma
> > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
> > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
>
> Don't mention --param in user documentation directly.
That brings up a question we discussed in internal reviews: should
this even be a -fwidening-mul-defer-fma flag then — or just another
--param?
After all, this is the companion for a --param, so the alternate
opinion was to have a second --param that would be set to either 0 or
1. However, such a param (that will always be set either to 0 or 1)
seems odd.
The third option discussed was a target-hook, but that doesn't allow a
commandline-override.
Any thoughts?
>
> > +
> > ftree-builtin-call-dce
> > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
> > Enable conditional dead code elimination for builtin calls.
> > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def b/gcc/config/aarch64/aarch64-tuning-flags.def
> > index 058dadecccaa..5b087bfc9f03 100644
> > --- a/gcc/config/aarch64/aarch64-tuning-flags.def
> > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def
> > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
> >
> > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
> >
> > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set
> > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
> > + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
> > + pass's FMA deferring. */
> > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
> >
> > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
> > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > index 78f1eae8336c..fdb5bbbdfbce 100644
> > --- a/gcc/config/aarch64/aarch64.cc
> > +++ b/gcc/config/aarch64/aarch64.cc
> > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
> > && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
> > opts->x_flag_prefetch_loop_arrays = 1;
> >
> > - /* Avoid loop-dependant FMA chains. */
> > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
> > + keeps using --param=avoid-fma-max-bits; the widening-mul-side
> > + deferring is gated separately by -fwidening-mul-defer-fma, so we
> > + suppress only the deferring on these cores while leaving the reassoc
> > + reorder active. */
> > if (aarch64_tune_params.extra_tuning_flags
> > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
> > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
> > - 512);
> > + {
> > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
> > + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
> > + }
> >
> > /* Consider fully pipelined FMA in reassociation. */
> > if (aarch64_tune_params.extra_tuning_flags
> > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> > index 208625cafc55..f39902f77e9b 100644
> > --- a/gcc/doc/invoke.texi
> > +++ b/gcc/doc/invoke.texi
> > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
> > -funconstrained-commons -funit-at-a-time -funroll-all-loops
> > -funroll-loops -funsafe-math-optimizations -funswitch-loops
> > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
> > --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
> > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
> > +-fuse-linker-plugin -fzero-call-used-regs
> > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
> >
> > @item Program Instrumentation Options
> > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
> >
> > Enabled by default with @option{-funroll-loops}.
> >
> > +@opindex fwidening-mul-defer-fma
> > +@opindex fno-widening-mul-defer-fma
> > +@item -fwidening-mul-defer-fma
> > +When FMA avoidance is requested for loop-carried reductions (that is, when
> > +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
> > +refuses to contract a multiply-add whose result feeds a loop-header PHI,
> > +leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
> > +to suppress this deferring and contract such reductions into an FMA, while
> > +still leaving @option{--param=avoid-fma-max-bits} in effect for the
> > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
> > +This is useful on targets whose pipeline disfavours the extra floating-point
> > +operation more than the loop-carried FMA latency. The option is enabled by
> > +default.
> Here either.
>
>
> > +
> > @opindex fwhole-program
> > @opindex fno-whole-program
> > @item -fwhole-program
> > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > new file mode 100644
> > index 000000000000..b542274eb763
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > @@ -0,0 +1,19 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
> > +
> > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
> > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
> > + reduction is contracted to fmadd rather than deferred to fmul + fadd.
> > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
> > +
> > +double
> > +dot (const double *a, const double *b, int n)
> > +{
> > + double s = 0.0;
> > + for (int i = 0; i < n; i++)
> > + s += a[i] * b[i];
> > + return s;
> > +}
> > +
> > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > new file mode 100644
> > index 000000000000..764dd2457435
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > @@ -0,0 +1,21 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
> > +
> > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
> > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
> > + whose result feeds the loop-header phi: the reduction stays as a
> > + separate fmul + fadd. This is the behaviour the new flag decouples
> > + from the reassoc reorder (also gated by avoid-fma-max-bits). */
> > +
> > +double
> > +dot (const double *a, const double *b, int n)
> > +{
> > + double s = 0.0;
> > + for (int i = 0; i < n; i++)
> > + s += a[i] * b[i];
> > + return s;
> > +}
> > +
> > +/* { dg-final { scan-assembler {\tfmul\t} } } */
> > +/* { dg-final { scan-assembler {\tfadd\t} } } */
> > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > new file mode 100644
> > index 000000000000..5e041afc18fc
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > @@ -0,0 +1,20 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
> > +
> > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
> > + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
> > + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
> > + so the loop-carried multiply-add is contracted to a single fmadd. This
> > + proves the new flag decouples the deferring from avoid-fma-max-bits. */
> > +
> > +double
> > +dot (const double *a, const double *b, int n)
> > +{
> > + double s = 0.0;
> > + for (int i = 0; i < n; i++)
> > + s += a[i] * b[i];
> > + return s;
> > +}
> > +
> > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
> > index f0ede668d95e..a00146922713 100644
> > --- a/gcc/tree-ssa-math-opts.cc
> > +++ b/gcc/tree-ssa-math-opts.cc
> > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
> > {
> > gimple_stmt_iterator gsi;
> >
> > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
> > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
> > + && flag_widening_mul_defer_fma);
> >
> > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
> > psi = psi_next)
> > --
> > 2.34.1
> >
> > base-commit: 3b443e764144526748d2b51d017d745f1956377e
> > branch: ptomsich/475-widening-mul-defer-fma
On Tue, Jul 7, 2026 at 2:46 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
>
> On Tue, 7 Jul 2026 at 23:40, Andrea Pinski
> <andrew.pinski@oss.qualcomm.com> wrote:
> >
> > On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> > >
> > > param_avoid_fma_max_bits gates two independent transforms: the
> > > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
> > > loop-carried multiply-add as fmul + fadd, and the reassoc
> > > loop-carried-FMA reorder added in r14-5779-g746344dd538
> > > (PR tree-optimization/110279), which parallelises 3+ operand chains.
> > > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
> > > type they switch on at the same threshold and a target cannot keep one
> > > while dropping the other.
> > >
> > > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
> > > family): a 2-operand reduction such as an sgemm inner K-loop is left as
> > > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
> > > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
> > >
> > > Add a generic -fwidening-mul-defer-fma (default on) that gates only the
> > > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
> > > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
> > > target and no default behaviour changes.
> >
> > Note this is just a review on the documentation and nothing else.
> >
> > >
> > > gcc/ChangeLog:
> > >
> > > * common.opt (fwidening-mul-defer-fma): New option.
> > > * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
> > > Gate fma_deferring_state's enable predicate on
> > > flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
> > > * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
> > > Update the comment for the -fwidening-mul-defer-fma effect.
> > > * config/aarch64/aarch64.cc (aarch64_override_options_internal):
> > > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
> > > flag_widening_mul_defer_fma.
> > > * doc/invoke.texi (-fwidening-mul-defer-fma): Document.
> > >
> > > gcc/testsuite/ChangeLog:
> > >
> > > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
> > > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
> > > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
> > >
> > > Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
> > > ---
> > >
> > > gcc/common.opt | 4 ++++
> > > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
> > > gcc/config/aarch64/aarch64.cc | 12 ++++++++---
> > > gcc/doc/invoke.texi | 17 ++++++++++++++-
> > > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
> > > .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
> > > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
> > > gcc/tree-ssa-math-opts.cc | 3 ++-
> > > 8 files changed, 95 insertions(+), 5 deletions(-)
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > >
> > > diff --git a/gcc/common.opt b/gcc/common.opt
> > > index 218dddf5dfe9..9be7dd6b2315 100644
> > > --- a/gcc/common.opt
> > > +++ b/gcc/common.opt
> > > @@ -3639,6 +3639,10 @@ fweb
> > > Common Var(flag_web) Optimization EnabledBy(funroll-loops)
> > > Construct webs and split unrelated uses of single variable.
> > >
> > > +fwidening-mul-defer-fma
> > > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
> > > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
> >
> > Don't mention --param in user documentation directly.
>
> That brings up a question we discussed in internal reviews: should
> this even be a -fwidening-mul-defer-fma flag then — or just another
> --param?
> After all, this is the companion for a --param, so the alternate
> opinion was to have a second --param that would be set to either 0 or
> 1. However, such a param (that will always be set either to 0 or 1)
> seems odd.
There are plenty of params that are already only support 0 or 1.
Just after the line 34 of params.opt 4 in a row:
-param=asan-globals=
Common Joined UInteger Var(param_asan_globals) Init(1) IntegerRange(0, 1) Param
Enable asan globals protection.
And fully-pipelined-fma is one too:
-param=fully-pipelined-fma=
Common Joined UInteger Var(param_fully_pipelined_fma) Init(0)
IntegerRange(0, 1) Param Optimization
Whether the target fully pipelines FMA instructions. If non-zero,
reassociation considers the benefit of parallelizing FMA's multiplication
part and addition part, assuming FMUL and FMA use the same units that can
also do FADD.
>
> The third option discussed was a target-hook, but that doesn't allow a
> commandline-override.
>
> Any thoughts?
>
> >
> > > +
> > > ftree-builtin-call-dce
> > > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
> > > Enable conditional dead code elimination for builtin calls.
> > > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > index 058dadecccaa..5b087bfc9f03 100644
> > > --- a/gcc/config/aarch64/aarch64-tuning-flags.def
> > > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
> > >
> > > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
> > >
> > > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set
> > > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
> > > + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
> > > + pass's FMA deferring. */
> > > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
> > >
> > > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
> > > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > > index 78f1eae8336c..fdb5bbbdfbce 100644
> > > --- a/gcc/config/aarch64/aarch64.cc
> > > +++ b/gcc/config/aarch64/aarch64.cc
> > > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
> > > && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
> > > opts->x_flag_prefetch_loop_arrays = 1;
> > >
> > > - /* Avoid loop-dependant FMA chains. */
> > > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
> > > + keeps using --param=avoid-fma-max-bits; the widening-mul-side
> > > + deferring is gated separately by -fwidening-mul-defer-fma, so we
> > > + suppress only the deferring on these cores while leaving the reassoc
> > > + reorder active. */
> > > if (aarch64_tune_params.extra_tuning_flags
> > > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
> > > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
> > > - 512);
> > > + {
> > > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
> > > + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
> > > + }
> > >
> > > /* Consider fully pipelined FMA in reassociation. */
> > > if (aarch64_tune_params.extra_tuning_flags
> > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> > > index 208625cafc55..f39902f77e9b 100644
> > > --- a/gcc/doc/invoke.texi
> > > +++ b/gcc/doc/invoke.texi
> > > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
> > > -funconstrained-commons -funit-at-a-time -funroll-all-loops
> > > -funroll-loops -funsafe-math-optimizations -funswitch-loops
> > > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
> > > --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
> > > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
> > > +-fuse-linker-plugin -fzero-call-used-regs
> > > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
> > >
> > > @item Program Instrumentation Options
> > > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
> > >
> > > Enabled by default with @option{-funroll-loops}.
> > >
> > > +@opindex fwidening-mul-defer-fma
> > > +@opindex fno-widening-mul-defer-fma
> > > +@item -fwidening-mul-defer-fma
> > > +When FMA avoidance is requested for loop-carried reductions (that is, when
> > > +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
> > > +refuses to contract a multiply-add whose result feeds a loop-header PHI,
> > > +leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
> > > +to suppress this deferring and contract such reductions into an FMA, while
> > > +still leaving @option{--param=avoid-fma-max-bits} in effect for the
> > > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
> > > +This is useful on targets whose pipeline disfavours the extra floating-point
> > > +operation more than the loop-carried FMA latency. The option is enabled by
> > > +default.
> > Here either.
> >
> >
> > > +
> > > @opindex fwhole-program
> > > @opindex fno-whole-program
> > > @item -fwhole-program
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > new file mode 100644
> > > index 000000000000..b542274eb763
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > @@ -0,0 +1,19 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
> > > +
> > > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
> > > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
> > > + reduction is contracted to fmadd rather than deferred to fmul + fadd.
> > > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
> > > +
> > > +double
> > > +dot (const double *a, const double *b, int n)
> > > +{
> > > + double s = 0.0;
> > > + for (int i = 0; i < n; i++)
> > > + s += a[i] * b[i];
> > > + return s;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > new file mode 100644
> > > index 000000000000..764dd2457435
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > @@ -0,0 +1,21 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
> > > +
> > > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
> > > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
> > > + whose result feeds the loop-header phi: the reduction stays as a
> > > + separate fmul + fadd. This is the behaviour the new flag decouples
> > > + from the reassoc reorder (also gated by avoid-fma-max-bits). */
> > > +
> > > +double
> > > +dot (const double *a, const double *b, int n)
> > > +{
> > > + double s = 0.0;
> > > + for (int i = 0; i < n; i++)
> > > + s += a[i] * b[i];
> > > + return s;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler {\tfmul\t} } } */
> > > +/* { dg-final { scan-assembler {\tfadd\t} } } */
> > > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
> > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > new file mode 100644
> > > index 000000000000..5e041afc18fc
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > @@ -0,0 +1,20 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
> > > +
> > > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
> > > + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
> > > + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
> > > + so the loop-carried multiply-add is contracted to a single fmadd. This
> > > + proves the new flag decouples the deferring from avoid-fma-max-bits. */
> > > +
> > > +double
> > > +dot (const double *a, const double *b, int n)
> > > +{
> > > + double s = 0.0;
> > > + for (int i = 0; i < n; i++)
> > > + s += a[i] * b[i];
> > > + return s;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
> > > index f0ede668d95e..a00146922713 100644
> > > --- a/gcc/tree-ssa-math-opts.cc
> > > +++ b/gcc/tree-ssa-math-opts.cc
> > > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
> > > {
> > > gimple_stmt_iterator gsi;
> > >
> > > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
> > > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
> > > + && flag_widening_mul_defer_fma);
> > >
> > > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
> > > psi = psi_next)
> > > --
> > > 2.34.1
> > >
> > > base-commit: 3b443e764144526748d2b51d017d745f1956377e
> > > branch: ptomsich/475-widening-mul-defer-fma
On Tue, 7 Jul 2026 at 23:51, Andrea Pinski
<andrew.pinski@oss.qualcomm.com> wrote:
>
> On Tue, Jul 7, 2026 at 2:46 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> >
> > On Tue, 7 Jul 2026 at 23:40, Andrea Pinski
> > <andrew.pinski@oss.qualcomm.com> wrote:
> > >
> > > On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> > > >
> > > > param_avoid_fma_max_bits gates two independent transforms: the
> > > > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
> > > > loop-carried multiply-add as fmul + fadd, and the reassoc
> > > > loop-carried-FMA reorder added in r14-5779-g746344dd538
> > > > (PR tree-optimization/110279), which parallelises 3+ operand chains.
> > > > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
> > > > type they switch on at the same threshold and a target cannot keep one
> > > > while dropping the other.
> > > >
> > > > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
> > > > family): a 2-operand reduction such as an sgemm inner K-loop is left as
> > > > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
> > > > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
> > > >
> > > > Add a generic -fwidening-mul-defer-fma (default on) that gates only the
> > > > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
> > > > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
> > > > target and no default behaviour changes.
> > >
> > > Note this is just a review on the documentation and nothing else.
> > >
> > > >
> > > > gcc/ChangeLog:
> > > >
> > > > * common.opt (fwidening-mul-defer-fma): New option.
> > > > * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
> > > > Gate fma_deferring_state's enable predicate on
> > > > flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
> > > > * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
> > > > Update the comment for the -fwidening-mul-defer-fma effect.
> > > > * config/aarch64/aarch64.cc (aarch64_override_options_internal):
> > > > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
> > > > flag_widening_mul_defer_fma.
> > > > * doc/invoke.texi (-fwidening-mul-defer-fma): Document.
> > > >
> > > > gcc/testsuite/ChangeLog:
> > > >
> > > > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
> > > > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
> > > > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
> > > >
> > > > Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
> > > > ---
> > > >
> > > > gcc/common.opt | 4 ++++
> > > > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
> > > > gcc/config/aarch64/aarch64.cc | 12 ++++++++---
> > > > gcc/doc/invoke.texi | 17 ++++++++++++++-
> > > > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
> > > > .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
> > > > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
> > > > gcc/tree-ssa-math-opts.cc | 3 ++-
> > > > 8 files changed, 95 insertions(+), 5 deletions(-)
> > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > >
> > > > diff --git a/gcc/common.opt b/gcc/common.opt
> > > > index 218dddf5dfe9..9be7dd6b2315 100644
> > > > --- a/gcc/common.opt
> > > > +++ b/gcc/common.opt
> > > > @@ -3639,6 +3639,10 @@ fweb
> > > > Common Var(flag_web) Optimization EnabledBy(funroll-loops)
> > > > Construct webs and split unrelated uses of single variable.
> > > >
> > > > +fwidening-mul-defer-fma
> > > > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
> > > > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
> > >
> > > Don't mention --param in user documentation directly.
> >
> > That brings up a question we discussed in internal reviews: should
> > this even be a -fwidening-mul-defer-fma flag then — or just another
> > --param?
> > After all, this is the companion for a --param, so the alternate
> > opinion was to have a second --param that would be set to either 0 or
> > 1. However, such a param (that will always be set either to 0 or 1)
> > seems odd.
>
>
> There are plenty of params that are already only support 0 or 1.
> Just after the line 34 of params.opt 4 in a row:
> -param=asan-globals=
> Common Joined UInteger Var(param_asan_globals) Init(1) IntegerRange(0, 1) Param
> Enable asan globals protection.
>
> And fully-pipelined-fma is one too:
> -param=fully-pipelined-fma=
> Common Joined UInteger Var(param_fully_pipelined_fma) Init(0)
> IntegerRange(0, 1) Param Optimization
> Whether the target fully pipelines FMA instructions. If non-zero,
> reassociation considers the benefit of parallelizing FMA's multiplication
> part and addition part, assuming FMUL and FMA use the same units that can
> also do FADD.
I'll take this as "follow the established precedent, even if you don't
find it pretty".
ACK.
>
> >
> > The third option discussed was a target-hook, but that doesn't allow a
> > commandline-override.
> >
> > Any thoughts?
> >
> > >
> > > > +
> > > > ftree-builtin-call-dce
> > > > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
> > > > Enable conditional dead code elimination for builtin calls.
> > > > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > index 058dadecccaa..5b087bfc9f03 100644
> > > > --- a/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
> > > >
> > > > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
> > > >
> > > > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set
> > > > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
> > > > + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
> > > > + pass's FMA deferring. */
> > > > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
> > > >
> > > > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
> > > > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > > > index 78f1eae8336c..fdb5bbbdfbce 100644
> > > > --- a/gcc/config/aarch64/aarch64.cc
> > > > +++ b/gcc/config/aarch64/aarch64.cc
> > > > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
> > > > && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
> > > > opts->x_flag_prefetch_loop_arrays = 1;
> > > >
> > > > - /* Avoid loop-dependant FMA chains. */
> > > > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
> > > > + keeps using --param=avoid-fma-max-bits; the widening-mul-side
> > > > + deferring is gated separately by -fwidening-mul-defer-fma, so we
> > > > + suppress only the deferring on these cores while leaving the reassoc
> > > > + reorder active. */
> > > > if (aarch64_tune_params.extra_tuning_flags
> > > > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
> > > > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
> > > > - 512);
> > > > + {
> > > > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
> > > > + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
> > > > + }
> > > >
> > > > /* Consider fully pipelined FMA in reassociation. */
> > > > if (aarch64_tune_params.extra_tuning_flags
> > > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> > > > index 208625cafc55..f39902f77e9b 100644
> > > > --- a/gcc/doc/invoke.texi
> > > > +++ b/gcc/doc/invoke.texi
> > > > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
> > > > -funconstrained-commons -funit-at-a-time -funroll-all-loops
> > > > -funroll-loops -funsafe-math-optimizations -funswitch-loops
> > > > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
> > > > --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
> > > > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
> > > > +-fuse-linker-plugin -fzero-call-used-regs
> > > > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
> > > >
> > > > @item Program Instrumentation Options
> > > > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
> > > >
> > > > Enabled by default with @option{-funroll-loops}.
> > > >
> > > > +@opindex fwidening-mul-defer-fma
> > > > +@opindex fno-widening-mul-defer-fma
> > > > +@item -fwidening-mul-defer-fma
> > > > +When FMA avoidance is requested for loop-carried reductions (that is, when
> > > > +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
> > > > +refuses to contract a multiply-add whose result feeds a loop-header PHI,
> > > > +leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
> > > > +to suppress this deferring and contract such reductions into an FMA, while
> > > > +still leaving @option{--param=avoid-fma-max-bits} in effect for the
> > > > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
> > > > +This is useful on targets whose pipeline disfavours the extra floating-point
> > > > +operation more than the loop-carried FMA latency. The option is enabled by
> > > > +default.
> > > Here either.
> > >
> > >
> > > > +
> > > > @opindex fwhole-program
> > > > @opindex fno-whole-program
> > > > @item -fwhole-program
> > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > new file mode 100644
> > > > index 000000000000..b542274eb763
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > @@ -0,0 +1,19 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
> > > > +
> > > > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
> > > > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
> > > > + reduction is contracted to fmadd rather than deferred to fmul + fadd.
> > > > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
> > > > +
> > > > +double
> > > > +dot (const double *a, const double *b, int n)
> > > > +{
> > > > + double s = 0.0;
> > > > + for (int i = 0; i < n; i++)
> > > > + s += a[i] * b[i];
> > > > + return s;
> > > > +}
> > > > +
> > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > new file mode 100644
> > > > index 000000000000..764dd2457435
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > @@ -0,0 +1,21 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
> > > > +
> > > > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
> > > > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
> > > > + whose result feeds the loop-header phi: the reduction stays as a
> > > > + separate fmul + fadd. This is the behaviour the new flag decouples
> > > > + from the reassoc reorder (also gated by avoid-fma-max-bits). */
> > > > +
> > > > +double
> > > > +dot (const double *a, const double *b, int n)
> > > > +{
> > > > + double s = 0.0;
> > > > + for (int i = 0; i < n; i++)
> > > > + s += a[i] * b[i];
> > > > + return s;
> > > > +}
> > > > +
> > > > +/* { dg-final { scan-assembler {\tfmul\t} } } */
> > > > +/* { dg-final { scan-assembler {\tfadd\t} } } */
> > > > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
> > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > > new file mode 100644
> > > > index 000000000000..5e041afc18fc
> > > > --- /dev/null
> > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > > @@ -0,0 +1,20 @@
> > > > +/* { dg-do compile } */
> > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
> > > > +
> > > > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
> > > > + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
> > > > + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
> > > > + so the loop-carried multiply-add is contracted to a single fmadd. This
> > > > + proves the new flag decouples the deferring from avoid-fma-max-bits. */
> > > > +
> > > > +double
> > > > +dot (const double *a, const double *b, int n)
> > > > +{
> > > > + double s = 0.0;
> > > > + for (int i = 0; i < n; i++)
> > > > + s += a[i] * b[i];
> > > > + return s;
> > > > +}
> > > > +
> > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
> > > > index f0ede668d95e..a00146922713 100644
> > > > --- a/gcc/tree-ssa-math-opts.cc
> > > > +++ b/gcc/tree-ssa-math-opts.cc
> > > > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
> > > > {
> > > > gimple_stmt_iterator gsi;
> > > >
> > > > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
> > > > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
> > > > + && flag_widening_mul_defer_fma);
> > > >
> > > > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
> > > > psi = psi_next)
> > > > --
> > > > 2.34.1
> > > >
> > > > base-commit: 3b443e764144526748d2b51d017d745f1956377e
> > > > branch: ptomsich/475-widening-mul-defer-fma
Superseded by v2 at
https://gcc.gnu.org/pipermail/gcc-patches/2026-July/723165.html
On Tue, 7 Jul 2026 at 23:53, Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
>
> On Tue, 7 Jul 2026 at 23:51, Andrea Pinski
> <andrew.pinski@oss.qualcomm.com> wrote:
> >
> > On Tue, Jul 7, 2026 at 2:46 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> > >
> > > On Tue, 7 Jul 2026 at 23:40, Andrea Pinski
> > > <andrew.pinski@oss.qualcomm.com> wrote:
> > > >
> > > > On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <philipp.tomsich@vrull.eu> wrote:
> > > > >
> > > > > param_avoid_fma_max_bits gates two independent transforms: the
> > > > > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
> > > > > loop-carried multiply-add as fmul + fadd, and the reassoc
> > > > > loop-carried-FMA reorder added in r14-5779-g746344dd538
> > > > > (PR tree-optimization/110279), which parallelises 3+ operand chains.
> > > > > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
> > > > > type they switch on at the same threshold and a target cannot keep one
> > > > > while dropping the other.
> > > > >
> > > > > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
> > > > > family): a 2-operand reduction such as an sgemm inner K-loop is left as
> > > > > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
> > > > > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
> > > > >
> > > > > Add a generic -fwidening-mul-defer-fma (default on) that gates only the
> > > > > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
> > > > > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
> > > > > target and no default behaviour changes.
> > > >
> > > > Note this is just a review on the documentation and nothing else.
> > > >
> > > > >
> > > > > gcc/ChangeLog:
> > > > >
> > > > > * common.opt (fwidening-mul-defer-fma): New option.
> > > > > * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
> > > > > Gate fma_deferring_state's enable predicate on
> > > > > flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
> > > > > * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
> > > > > Update the comment for the -fwidening-mul-defer-fma effect.
> > > > > * config/aarch64/aarch64.cc (aarch64_override_options_internal):
> > > > > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
> > > > > flag_widening_mul_defer_fma.
> > > > > * doc/invoke.texi (-fwidening-mul-defer-fma): Document.
> > > > >
> > > > > gcc/testsuite/ChangeLog:
> > > > >
> > > > > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
> > > > > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
> > > > > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
> > > > >
> > > > > Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
> > > > > ---
> > > > >
> > > > > gcc/common.opt | 4 ++++
> > > > > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
> > > > > gcc/config/aarch64/aarch64.cc | 12 ++++++++---
> > > > > gcc/doc/invoke.texi | 17 ++++++++++++++-
> > > > > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
> > > > > .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
> > > > > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
> > > > > gcc/tree-ssa-math-opts.cc | 3 ++-
> > > > > 8 files changed, 95 insertions(+), 5 deletions(-)
> > > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > > create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > > >
> > > > > diff --git a/gcc/common.opt b/gcc/common.opt
> > > > > index 218dddf5dfe9..9be7dd6b2315 100644
> > > > > --- a/gcc/common.opt
> > > > > +++ b/gcc/common.opt
> > > > > @@ -3639,6 +3639,10 @@ fweb
> > > > > Common Var(flag_web) Optimization EnabledBy(funroll-loops)
> > > > > Construct webs and split unrelated uses of single variable.
> > > > >
> > > > > +fwidening-mul-defer-fma
> > > > > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
> > > > > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
> > > >
> > > > Don't mention --param in user documentation directly.
> > >
> > > That brings up a question we discussed in internal reviews: should
> > > this even be a -fwidening-mul-defer-fma flag then — or just another
> > > --param?
> > > After all, this is the companion for a --param, so the alternate
> > > opinion was to have a second --param that would be set to either 0 or
> > > 1. However, such a param (that will always be set either to 0 or 1)
> > > seems odd.
> >
> >
> > There are plenty of params that are already only support 0 or 1.
> > Just after the line 34 of params.opt 4 in a row:
> > -param=asan-globals=
> > Common Joined UInteger Var(param_asan_globals) Init(1) IntegerRange(0, 1) Param
> > Enable asan globals protection.
> >
> > And fully-pipelined-fma is one too:
> > -param=fully-pipelined-fma=
> > Common Joined UInteger Var(param_fully_pipelined_fma) Init(0)
> > IntegerRange(0, 1) Param Optimization
> > Whether the target fully pipelines FMA instructions. If non-zero,
> > reassociation considers the benefit of parallelizing FMA's multiplication
> > part and addition part, assuming FMUL and FMA use the same units that can
> > also do FADD.
>
> I'll take this as "follow the established precedent, even if you don't
> find it pretty".
> ACK.
>
> >
> > >
> > > The third option discussed was a target-hook, but that doesn't allow a
> > > commandline-override.
> > >
> > > Any thoughts?
> > >
> > > >
> > > > > +
> > > > > ftree-builtin-call-dce
> > > > > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
> > > > > Enable conditional dead code elimination for builtin calls.
> > > > > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > > index 058dadecccaa..5b087bfc9f03 100644
> > > > > --- a/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def
> > > > > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
> > > > >
> > > > > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
> > > > >
> > > > > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set
> > > > > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
> > > > > + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
> > > > > + pass's FMA deferring. */
> > > > > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
> > > > >
> > > > > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
> > > > > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > > > > index 78f1eae8336c..fdb5bbbdfbce 100644
> > > > > --- a/gcc/config/aarch64/aarch64.cc
> > > > > +++ b/gcc/config/aarch64/aarch64.cc
> > > > > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
> > > > > && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
> > > > > opts->x_flag_prefetch_loop_arrays = 1;
> > > > >
> > > > > - /* Avoid loop-dependant FMA chains. */
> > > > > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
> > > > > + keeps using --param=avoid-fma-max-bits; the widening-mul-side
> > > > > + deferring is gated separately by -fwidening-mul-defer-fma, so we
> > > > > + suppress only the deferring on these cores while leaving the reassoc
> > > > > + reorder active. */
> > > > > if (aarch64_tune_params.extra_tuning_flags
> > > > > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
> > > > > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
> > > > > - 512);
> > > > > + {
> > > > > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
> > > > > + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
> > > > > + }
> > > > >
> > > > > /* Consider fully pipelined FMA in reassociation. */
> > > > > if (aarch64_tune_params.extra_tuning_flags
> > > > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> > > > > index 208625cafc55..f39902f77e9b 100644
> > > > > --- a/gcc/doc/invoke.texi
> > > > > +++ b/gcc/doc/invoke.texi
> > > > > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
> > > > > -funconstrained-commons -funit-at-a-time -funroll-all-loops
> > > > > -funroll-loops -funsafe-math-optimizations -funswitch-loops
> > > > > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
> > > > > --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
> > > > > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
> > > > > +-fuse-linker-plugin -fzero-call-used-regs
> > > > > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
> > > > >
> > > > > @item Program Instrumentation Options
> > > > > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
> > > > >
> > > > > Enabled by default with @option{-funroll-loops}.
> > > > >
> > > > > +@opindex fwidening-mul-defer-fma
> > > > > +@opindex fno-widening-mul-defer-fma
> > > > > +@item -fwidening-mul-defer-fma
> > > > > +When FMA avoidance is requested for loop-carried reductions (that is, when
> > > > > +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
> > > > > +refuses to contract a multiply-add whose result feeds a loop-header PHI,
> > > > > +leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
> > > > > +to suppress this deferring and contract such reductions into an FMA, while
> > > > > +still leaving @option{--param=avoid-fma-max-bits} in effect for the
> > > > > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
> > > > > +This is useful on targets whose pipeline disfavours the extra floating-point
> > > > > +operation more than the loop-carried FMA latency. The option is enabled by
> > > > > +default.
> > > > Here either.
> > > >
> > > >
> > > > > +
> > > > > @opindex fwhole-program
> > > > > @opindex fno-whole-program
> > > > > @item -fwhole-program
> > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > > new file mode 100644
> > > > > index 000000000000..b542274eb763
> > > > > --- /dev/null
> > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
> > > > > @@ -0,0 +1,19 @@
> > > > > +/* { dg-do compile } */
> > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
> > > > > +
> > > > > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
> > > > > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
> > > > > + reduction is contracted to fmadd rather than deferred to fmul + fadd.
> > > > > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
> > > > > +
> > > > > +double
> > > > > +dot (const double *a, const double *b, int n)
> > > > > +{
> > > > > + double s = 0.0;
> > > > > + for (int i = 0; i < n; i++)
> > > > > + s += a[i] * b[i];
> > > > > + return s;
> > > > > +}
> > > > > +
> > > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > > new file mode 100644
> > > > > index 000000000000..764dd2457435
> > > > > --- /dev/null
> > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
> > > > > @@ -0,0 +1,21 @@
> > > > > +/* { dg-do compile } */
> > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
> > > > > +
> > > > > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
> > > > > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
> > > > > + whose result feeds the loop-header phi: the reduction stays as a
> > > > > + separate fmul + fadd. This is the behaviour the new flag decouples
> > > > > + from the reassoc reorder (also gated by avoid-fma-max-bits). */
> > > > > +
> > > > > +double
> > > > > +dot (const double *a, const double *b, int n)
> > > > > +{
> > > > > + double s = 0.0;
> > > > > + for (int i = 0; i < n; i++)
> > > > > + s += a[i] * b[i];
> > > > > + return s;
> > > > > +}
> > > > > +
> > > > > +/* { dg-final { scan-assembler {\tfmul\t} } } */
> > > > > +/* { dg-final { scan-assembler {\tfadd\t} } } */
> > > > > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
> > > > > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > > > new file mode 100644
> > > > > index 000000000000..5e041afc18fc
> > > > > --- /dev/null
> > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
> > > > > @@ -0,0 +1,20 @@
> > > > > +/* { dg-do compile } */
> > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
> > > > > +
> > > > > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
> > > > > + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
> > > > > + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
> > > > > + so the loop-carried multiply-add is contracted to a single fmadd. This
> > > > > + proves the new flag decouples the deferring from avoid-fma-max-bits. */
> > > > > +
> > > > > +double
> > > > > +dot (const double *a, const double *b, int n)
> > > > > +{
> > > > > + double s = 0.0;
> > > > > + for (int i = 0; i < n; i++)
> > > > > + s += a[i] * b[i];
> > > > > + return s;
> > > > > +}
> > > > > +
> > > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */
> > > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */
> > > > > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
> > > > > index f0ede668d95e..a00146922713 100644
> > > > > --- a/gcc/tree-ssa-math-opts.cc
> > > > > +++ b/gcc/tree-ssa-math-opts.cc
> > > > > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
> > > > > {
> > > > > gimple_stmt_iterator gsi;
> > > > >
> > > > > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
> > > > > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
> > > > > + && flag_widening_mul_defer_fma);
> > > > >
> > > > > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
> > > > > psi = psi_next)
> > > > > --
> > > > > 2.34.1
> > > > >
> > > > > base-commit: 3b443e764144526748d2b51d017d745f1956377e
> > > > > branch: ptomsich/475-widening-mul-defer-fma
@@ -3639,6 +3639,10 @@ fweb
Common Var(flag_web) Optimization EnabledBy(funroll-loops)
Construct webs and split unrelated uses of single variable.
+fwidening-mul-defer-fma
+Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
+Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate multiply and add, when FMA avoidance is requested via --param=avoid-fma-max-bits.
+
ftree-builtin-call-dce
Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
Enable conditional dead code elimination for builtin calls.
@@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS)
AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", MATCHED_VECTOR_THROUGHPUT)
+/* For cores whose pipeline disfavours loop-carried serial FMAs: set
+ avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
+ chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
+ pass's FMA deferring. */
AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
@@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options *opts,
&& opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
opts->x_flag_prefetch_loop_arrays = 1;
- /* Avoid loop-dependant FMA chains. */
+ /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
+ keeps using --param=avoid-fma-max-bits; the widening-mul-side
+ deferring is gated separately by -fwidening-mul-defer-fma, so we
+ suppress only the deferring on these cores while leaving the reassoc
+ reorder active. */
if (aarch64_tune_params.extra_tuning_flags
& AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
- SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
- 512);
+ {
+ SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
+ SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
+ }
/* Consider fully pipelined FMA in reassociation. */
if (aarch64_tune_params.extra_tuning_flags
@@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
-funconstrained-commons -funit-at-a-time -funroll-all-loops
-funroll-loops -funsafe-math-optimizations -funswitch-loops
-fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
--fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
+-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
+-fuse-linker-plugin -fzero-call-used-regs
-O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
@item Program Instrumentation Options
@@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no longer stay in a
Enabled by default with @option{-funroll-loops}.
+@opindex fwidening-mul-defer-fma
+@opindex fno-widening-mul-defer-fma
+@item -fwidening-mul-defer-fma
+When FMA avoidance is requested for loop-carried reductions (that is, when
+@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
+refuses to contract a multiply-add whose result feeds a loop-header PHI,
+leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
+to suppress this deferring and contract such reductions into an FMA, while
+still leaving @option{--param=avoid-fma-max-bits} in effect for the
+@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
+This is useful on targets whose pipeline disfavours the extra floating-point
+operation more than the loop-carried FMA latency. The option is enabled by
+default.
+
@opindex fwhole-program
@opindex fno-whole-program
@item -fwhole-program
new file mode 100644
@@ -0,0 +1,19 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
+
+/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
+ clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
+ reduction is contracted to fmadd rather than deferred to fmul + fadd.
+ (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmul\t} } } */
new file mode 100644
@@ -0,0 +1,21 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512" } */
+
+/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
+ -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
+ whose result feeds the loop-header phi: the reduction stays as a
+ separate fmul + fadd. This is the behaviour the new flag decouples
+ from the reassoc reorder (also gated by avoid-fma-max-bits). */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmul\t} } } */
+/* { dg-final { scan-assembler {\tfadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
new file mode 100644
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
+
+/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
+ avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
+ but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
+ so the loop-carried multiply-add is contracted to a single fmadd. This
+ proves the new flag decouples the deferring from avoid-fma-max-bits. */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmul\t} } } */
@@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
{
gimple_stmt_iterator gsi;
- fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
+ fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
+ && flag_widening_mul_defer_fma);
for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
psi = psi_next)