[1/2] doc: describe the lane freedom of the widen_ssum and widen_usum patterns

  From: Kyrylo Tkachov <ktkachov@nvidia.com>

The documentation of widen_ssum and widen_usum says only that operand 1 is
added to operand 2, which reads as though each element of operand 1 has to
be accumulated into the element of operand 0 at the same position.  Nothing
says the pattern is only ever used for a reassociable reduction, so a port
cannot tell from the documentation whether it is allowed to regroup the
input elements, for example by using a pairwise widening add.

vect_recog_widen_sum_pattern only forms WIDEN_SUM_EXPR through
vect_reassociating_reduction_p, which requires the
statement to be a reduction, refuses a statement nested in the inner loop
of an outer-loop vectorization because the order of the computation matters
there, and refuses a type that needs a fold-left reduction.  The comment on
the pattern already describes the idiom as producing N/2 results by summing
up pairs of intermediate results.  The accumulator is only ever consumed by
a horizontal sum in the epilogue, so the distribution of input elements
over accumulator lanes is not observable.

Spell that out, and note the constraint that comes with it: an
implementation that adds elements before they reach the width of operand 0
has to widen them first so that no intermediate sum can overflow.

Is it okay to document this relaxation so that patch 2 can make use of
it in the aarch64 backend?

Thanks,
Kyrill

gcc/ChangeLog:

	* doc/md.texi (widen_ssum@var{n}@var{m}3, widen_usum@var{n}@var{m}3):
	Document that the assignment of input elements to accumulator
	elements is unconstrained, and fix a typo.

Signed-off-by: Kyrylo Tkachov <ktkachov@nvidia.com>
---
 gcc/doc/md.texi | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
  

On Tue, 4 Aug 2026, ktkachov@nvidia.com wrote:

> From: Kyrylo Tkachov <ktkachov@nvidia.com>
> 
> The documentation of widen_ssum and widen_usum says only that operand 1 is
> added to operand 2, which reads as though each element of operand 1 has to
> be accumulated into the element of operand 0 at the same position.  Nothing
> says the pattern is only ever used for a reassociable reduction, so a port
> cannot tell from the documentation whether it is allowed to regroup the
> input elements, for example by using a pairwise widening add.
> 
> vect_recog_widen_sum_pattern only forms WIDEN_SUM_EXPR through
> vect_reassociating_reduction_p, which requires the
> statement to be a reduction, refuses a statement nested in the inner loop
> of an outer-loop vectorization because the order of the computation matters
> there, and refuses a type that needs a fold-left reduction.  The comment on
> the pattern already describes the idiom as producing N/2 results by summing
> up pairs of intermediate results.  The accumulator is only ever consumed by
> a horizontal sum in the epilogue, so the distribution of input elements
> over accumulator lanes is not observable.
> 
> Spell that out, and note the constraint that comes with it: an
> implementation that adds elements before they reach the width of operand 0
> has to widen them first so that no intermediate sum can overflow.
> 
> Is it okay to document this relaxation so that patch 2 can make use of
> it in the aarch64 backend?

I wonder if this is a step in the correct direction, see PR67612 where
we want to apply a SLP pattern to exercise the actual lane mapping
done.

Do we want to have separate optabs for this?  IMO what you document
would be better named reduc_widen_ssum_optab (as opposed to
reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
variant would be expected to accumulate to a scalar (or lane zero),
the reduc_*_optab would be free in how to accumulate lanes and
the widen_ssum_optab would lay out exactly which source lanes are
summed to which destination lanes (I hope ISAs have matching behavior
here).

But sure, your patch documents the existing use.

Richard.


> Thanks,
> Kyrill
> 
> gcc/ChangeLog:
> 
> 	* doc/md.texi (widen_ssum@var{n}@var{m}3, widen_usum@var{n}@var{m}3):
> 	Document that the assignment of input elements to accumulator
> 	elements is unconstrained, and fix a typo.
> 
> Signed-off-by: Kyrylo Tkachov <ktkachov@nvidia.com>
> ---
>  gcc/doc/md.texi | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
> index c0026c17318..3235d456874 100644
> --- a/gcc/doc/md.texi
> +++ b/gcc/doc/md.texi
> @@ -5097,10 +5097,21 @@ in operand 0, which is of the same mode as operand 3.
>  @itemx @samp{widen_usum@var{n}@var{m}3}
>  Operands 0 and 2 are of the same mode, which is wider than the mode of
>  operand 1. Add operand 1 to operand 2 and place the widened result in
> -operand 0. (This is used express accumulation of elements into an accumulator
> -of a wider mode.)
> +operand 0. (This is used to express accumulation of elements into an
> +accumulator of a wider mode.)
>  @var{m} is the mode of operand 1 and @var{n} is the mode of operand 0.
>  
> +These patterns are only used for a reduction whose summation the vectorizer
> +has already established may be reassociated, and the accumulator is reduced
> +to a scalar by a horizontal sum once the loop is done.  An implementation is
> +therefore free to choose which elements of operand 1 it accumulates into
> +which element of operand 0, so long as every element of operand 1 is
> +accumulated exactly once.  In particular it may add adjacent elements of
> +operand 1 to each other before accumulating them, which is what a pairwise
> +widening add instruction does.  Doing so must not lose any value, so an
> +implementation that adds elements together before they reach the width of
> +operand 0 has to widen them enough for the intermediate sums to be exact.
> +
>  @mdindex smulhs@var{m}3
>  @mdindex umulhs@var{m}3
>  @item @samp{smulhs@var{m}3}
>
  

> I wonder if this is a step in the correct direction, see PR67612 where
> we want to apply a SLP pattern to exercise the actual lane mapping
> done.

> Do we want to have separate optabs for this?  IMO what you document
> would be better named reduc_widen_ssum_optab (as opposed to
> reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
> variant would be expected to accumulate to a scalar (or lane zero),
> the reduc_*_optab would be free in how to accumulate lanes and
> the widen_ssum_optab would lay out exactly which source lanes are
> summed to which destination lanes (I hope ISAs have matching behavior
> here).

I would be very much in favor of a separate optab rather than narrowing 
widen_[us]sum's scope.  You could argue that sum vs plus is already an 
inconsistency, though.
Last year I experimented with making widen_[us]sum available to RVV.  
That necessitated a few changes in the vectorizer (we don't expect 
"SLP-style" "number of lanes stays the same" everywhere) but it's not 
that big of a change.

Of course the general question remains where the widening should 
actually be recognized.  For riscv it works quite well doing it "late".
  

> -----Original Message-----
> From: Robin Dapp <rdapp.gcc@gmail.com>
> Sent: 04 August 2026 15:07
> To: Richard Biener <rguenther@suse.de>; Kyrylo Tkachov
> <ktkachov@nvidia.com>
> Cc: gcc-patches@gcc.gnu.org; Tamar Christina <Tamar.Christina@arm.com>;
> Robin Dapp <rdapp.gcc@gmail.com>
> Subject: Re: [PATCH 1/2] doc: describe the lane freedom of the widen_ssum
> and widen_usum patterns
> 
> > I wonder if this is a step in the correct direction, see PR67612 where
> > we want to apply a SLP pattern to exercise the actual lane mapping
> > done.
> 
> > Do we want to have separate optabs for this?  IMO what you document
> > would be better named reduc_widen_ssum_optab (as opposed to
> > reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
> > variant would be expected to accumulate to a scalar (or lane zero),
> > the reduc_*_optab would be free in how to accumulate lanes and
> > the widen_ssum_optab would lay out exactly which source lanes are
> > summed to which destination lanes (I hope ISAs have matching behavior
> > here).
> 
> I would be very much in favor of a separate optab rather than narrowing
> widen_[us]sum's scope.  You could argue that sum vs plus is already an
> inconsistency, though.
> Last year I experimented with making widen_[us]sum available to RVV.
> That necessitated a few changes in the vectorizer (we don't expect
> "SLP-style" "number of lanes stays the same" everywhere) but it's not
> that big of a change.
> 
> Of course the general question remains where the widening should
> actually be recognized.  For riscv it works quite well doing it "late".
> 

I think the widening and other similar optimizations need to stay in the vectorizer
because for masked based targets like SVE if we generate an unpacking then
the predicate needs to be unpacked too.

That means any "late" detection has to clean up the predicate operations too
which usually brings the chain out of what combine supports.  But also by
recognizing it In the vectorizer it allows us to cost alternate strategies like
unpacking or using sparse vectors.

So I don't think we can move this past costing as it has a significant effect on
Codegen.

Thanks,
Tamar

> --
> Regards
>  Robin
  

> On 4 Aug 2026, at 16:19, Tamar Christina <tamar.christina@arm.com> wrote:
> 
>> -----Original Message-----
>> From: Robin Dapp <rdapp.gcc@gmail.com>
>> Sent: 04 August 2026 15:07
>> To: Richard Biener <rguenther@suse.de>; Kyrylo Tkachov
>> <ktkachov@nvidia.com>
>> Cc: gcc-patches@gcc.gnu.org; Tamar Christina <Tamar.Christina@arm.com>;
>> Robin Dapp <rdapp.gcc@gmail.com>
>> Subject: Re: [PATCH 1/2] doc: describe the lane freedom of the widen_ssum
>> and widen_usum patterns
>> 
>>> I wonder if this is a step in the correct direction, see PR67612 where
>>> we want to apply a SLP pattern to exercise the actual lane mapping
>>> done.
>> 
>>> Do we want to have separate optabs for this?  IMO what you document
>>> would be better named reduc_widen_ssum_optab (as opposed to
>>> reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
>>> variant would be expected to accumulate to a scalar (or lane zero),
>>> the reduc_*_optab would be free in how to accumulate lanes and
>>> the widen_ssum_optab would lay out exactly which source lanes are
>>> summed to which destination lanes (I hope ISAs have matching behavior
>>> here).
>> 
>> I would be very much in favor of a separate optab rather than narrowing
>> widen_[us]sum's scope.  You could argue that sum vs plus is already an
>> inconsistency, though.
>> Last year I experimented with making widen_[us]sum available to RVV.
>> That necessitated a few changes in the vectorizer (we don't expect
>> "SLP-style" "number of lanes stays the same" everywhere) but it's not
>> that big of a change.
>> 
>> Of course the general question remains where the widening should
>> actually be recognized.  For riscv it works quite well doing it "late".
>> 
> 
> I think the widening and other similar optimizations need to stay in the vectorizer
> because for masked based targets like SVE if we generate an unpacking then
> the predicate needs to be unpacked too.
> 
> That means any "late" detection has to clean up the predicate operations too
> which usually brings the chain out of what combine supports.  But also by
> recognizing it In the vectorizer it allows us to cost alternate strategies like
> unpacking or using sparse vectors.
> 
> So I don't think we can move this past costing as it has a significant effect on
> Codegen.
> 

So as I’m not dealing day-to-day with the vectorizer, what would you recommend as the way forward here?
Rename the optabs currently used for lane-agnostic reduction to reduc_widen_[us]sum_optab and implement those for aarch64 as in patch 2/2 and document the widen_[us]sum optabs as lane-preserving?

Thanks,
Kyrill

> Thanks,
> Tamar
> 
>> --
>> Regards
>> Robin
  

On Wed, 5 Aug 2026, Kyrylo Tkachov wrote:

> 
> 
> > On 4 Aug 2026, at 16:19, Tamar Christina <tamar.christina@arm.com> wrote:
> > 
> >> -----Original Message-----
> >> From: Robin Dapp <rdapp.gcc@gmail.com>
> >> Sent: 04 August 2026 15:07
> >> To: Richard Biener <rguenther@suse.de>; Kyrylo Tkachov
> >> <ktkachov@nvidia.com>
> >> Cc: gcc-patches@gcc.gnu.org; Tamar Christina <Tamar.Christina@arm.com>;
> >> Robin Dapp <rdapp.gcc@gmail.com>
> >> Subject: Re: [PATCH 1/2] doc: describe the lane freedom of the widen_ssum
> >> and widen_usum patterns
> >> 
> >>> I wonder if this is a step in the correct direction, see PR67612 where
> >>> we want to apply a SLP pattern to exercise the actual lane mapping
> >>> done.
> >> 
> >>> Do we want to have separate optabs for this?  IMO what you document
> >>> would be better named reduc_widen_ssum_optab (as opposed to
> >>> reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
> >>> variant would be expected to accumulate to a scalar (or lane zero),
> >>> the reduc_*_optab would be free in how to accumulate lanes and
> >>> the widen_ssum_optab would lay out exactly which source lanes are
> >>> summed to which destination lanes (I hope ISAs have matching behavior
> >>> here).
> >> 
> >> I would be very much in favor of a separate optab rather than narrowing
> >> widen_[us]sum's scope.  You could argue that sum vs plus is already an
> >> inconsistency, though.
> >> Last year I experimented with making widen_[us]sum available to RVV.
> >> That necessitated a few changes in the vectorizer (we don't expect
> >> "SLP-style" "number of lanes stays the same" everywhere) but it's not
> >> that big of a change.
> >> 
> >> Of course the general question remains where the widening should
> >> actually be recognized.  For riscv it works quite well doing it "late".
> >> 
> > 
> > I think the widening and other similar optimizations need to stay in the vectorizer
> > because for masked based targets like SVE if we generate an unpacking then
> > the predicate needs to be unpacked too.
> > 
> > That means any "late" detection has to clean up the predicate operations too
> > which usually brings the chain out of what combine supports.  But also by
> > recognizing it In the vectorizer it allows us to cost alternate strategies like
> > unpacking or using sparse vectors.
> > 
> > So I don't think we can move this past costing as it has a significant effect on
> > Codegen.
> > 
> 
> So as I’m not dealing day-to-day with the vectorizer, what would you recommend as the way forward here?
> Rename the optabs currently used for lane-agnostic reduction to reduc_widen_[us]sum_optab and implement those for aarch64 as in patch 2/2 and document the widen_[us]sum optabs as lane-preserving?

naming consistency would be nice indeed.  Note WIDEN_SUM_EXPR
and related should also be unambiguously documented (I'm fine with
chosing lane-agnostic interpretations for them - in fact they should
all cease to exist and be replaced with internal functions only).

> Thanks,
> Kyrill
> 
> > Thanks,
> > Tamar
> > 
> >> --
> >> Regards
> >> Robin
> 
> 
>
  

> On 5 Aug 2026, at 14:41, Richard Biener <rguenther@suse.de> wrote:
> 
> On Wed, 5 Aug 2026, Kyrylo Tkachov wrote:
> 
>> 
>> 
>>> On 4 Aug 2026, at 16:19, Tamar Christina <tamar.christina@arm.com> wrote:
>>> 
>>>> -----Original Message-----
>>>> From: Robin Dapp <rdapp.gcc@gmail.com>
>>>> Sent: 04 August 2026 15:07
>>>> To: Richard Biener <rguenther@suse.de>; Kyrylo Tkachov
>>>> <ktkachov@nvidia.com>
>>>> Cc: gcc-patches@gcc.gnu.org; Tamar Christina <Tamar.Christina@arm.com>;
>>>> Robin Dapp <rdapp.gcc@gmail.com>
>>>> Subject: Re: [PATCH 1/2] doc: describe the lane freedom of the widen_ssum
>>>> and widen_usum patterns
>>>> 
>>>>> I wonder if this is a step in the correct direction, see PR67612 where
>>>>> we want to apply a SLP pattern to exercise the actual lane mapping
>>>>> done.
>>>> 
>>>>> Do we want to have separate optabs for this?  IMO what you document
>>>>> would be better named reduc_widen_ssum_optab (as opposed to
>>>>> reduc_widen_ssum_scal_optab or widen_ssum_optab).  Where the _scal
>>>>> variant would be expected to accumulate to a scalar (or lane zero),
>>>>> the reduc_*_optab would be free in how to accumulate lanes and
>>>>> the widen_ssum_optab would lay out exactly which source lanes are
>>>>> summed to which destination lanes (I hope ISAs have matching behavior
>>>>> here).
>>>> 
>>>> I would be very much in favor of a separate optab rather than narrowing
>>>> widen_[us]sum's scope.  You could argue that sum vs plus is already an
>>>> inconsistency, though.
>>>> Last year I experimented with making widen_[us]sum available to RVV.
>>>> That necessitated a few changes in the vectorizer (we don't expect
>>>> "SLP-style" "number of lanes stays the same" everywhere) but it's not
>>>> that big of a change.
>>>> 
>>>> Of course the general question remains where the widening should
>>>> actually be recognized.  For riscv it works quite well doing it "late".
>>>> 
>>> 
>>> I think the widening and other similar optimizations need to stay in the vectorizer
>>> because for masked based targets like SVE if we generate an unpacking then
>>> the predicate needs to be unpacked too.
>>> 
>>> That means any "late" detection has to clean up the predicate operations too
>>> which usually brings the chain out of what combine supports.  But also by
>>> recognizing it In the vectorizer it allows us to cost alternate strategies like
>>> unpacking or using sparse vectors.
>>> 
>>> So I don't think we can move this past costing as it has a significant effect on
>>> Codegen.
>>> 
>> 
>> So as I’m not dealing day-to-day with the vectorizer, what would you recommend as the way forward here?
>> Rename the optabs currently used for lane-agnostic reduction to reduc_widen_[us]sum_optab and implement those for aarch64 as in patch 2/2 and document the widen_[us]sum optabs as lane-preserving?
> 
> naming consistency would be nice indeed.  Note WIDEN_SUM_EXPR
> and related should also be unambiguously documented (I'm fine with
> chosing lane-agnostic interpretations for them - in fact they should
> all cease to exist and be replaced with internal functions only).

Something like:
https://gcc.gnu.org/pipermail/gcc-patches/2026-August/727035.html ?
Thanks,
Kyrill


> 
>> Thanks,
>> Kyrill
>> 
>>> Thanks,
>>> Tamar
>>> 
>>>> --
>>>> Regards
>>>> Robin
>> 
>> 
>> 
> 
> -- 
> Richard Biener <rguenther@suse.de>
> SUSE Software Solutions Germany GmbH,
> Frankenstrasse 146, 90461 Nuernberg, Germany;
> GF: Jochen Jaser, Andrew McDonald, Abhinav Puri; (HRB 36809, AG Nuernberg)