Message ID | 94r9n910-qsp7-778q-5ooq-pno71s1sqn6o@fhfr.qr |
---|---|
State | Committed |
Headers |
Return-Path: <gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org> X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0D2A73857416 for <patchwork@sourceware.org>; Mon, 13 Sep 2021 14:38:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0D2A73857416 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1631543911; bh=JfCOPPxT/3/Hyl04q/TvbeNoXDitCbTC005Pghgc1JY=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=WiBeyFhLlJTK/vWIgimR3O5vutMN7mGmnG4OkAyUnxcsKhMVK3X4jPa+0bi9N8fmv DY2YTovwyZs9yiBvvXds0+A57czYS/OWoNWKC1KMSNvaW+kPFK+hIKZJ3nhnBX1EZb zXMRPULHAZZuBe9MfsuOUzNejv16gQMudkFTuEh4= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id A5E113857413 for <gcc-patches@gcc.gnu.org>; Mon, 13 Sep 2021 14:37:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A5E113857413 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9AB271FFF3; Mon, 13 Sep 2021 14:37:44 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 75F8213AB9; Mon, 13 Sep 2021 14:37:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id AauWGzhiP2EGagAAMHmgww (envelope-from <rguenther@suse.de>); Mon, 13 Sep 2021 14:37:44 +0000 Date: Mon, 13 Sep 2021 16:37:43 +0200 (CEST) To: gcc-patches@gcc.gnu.org Subject: [PATCH] Maintain (mis-)alignment info in the first element of a group Message-ID: <94r9n910-qsp7-778q-5ooq-pno71s1sqn6o@fhfr.qr> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list <gcc-patches.gcc.gnu.org> List-Unsubscribe: <https://gcc.gnu.org/mailman/options/gcc-patches>, <mailto:gcc-patches-request@gcc.gnu.org?subject=unsubscribe> List-Archive: <https://gcc.gnu.org/pipermail/gcc-patches/> List-Post: <mailto:gcc-patches@gcc.gnu.org> List-Help: <mailto:gcc-patches-request@gcc.gnu.org?subject=help> List-Subscribe: <https://gcc.gnu.org/mailman/listinfo/gcc-patches>, <mailto:gcc-patches-request@gcc.gnu.org?subject=subscribe> From: Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> Reply-To: Richard Biener <rguenther@suse.de> Cc: richard.sandiford@arm.com Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" <gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org> |
Series |
Maintain (mis-)alignment info in the first element of a group
|
|
Commit Message
Richard Biener
Sept. 13, 2021, 2:37 p.m. UTC
This changes us to maintain and compute (mis-)alignment info for the first element of a group only rather than for each DR when doing interleaving and for the earliest, first, or first in the SLP node (or any pair or all three of those) when SLP vectorizing. For this to work out the easiest way I have changed the accessors DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to the first element rather than adjusting all callers. dr_misalignment is moved out-of-line and I'm not too fond of the poly-int dances there (any hints?), but basically we are now adjusting the first elements misalignment based on the DR_INIT difference. Bootstrap & regtest running on x86_64-unknown-linux-gnu. Richard. 2021-09-13 Richard Biener <rguenther@suse.de> * tree-vectorizer.h (dr_misalignment): Move out of line. (dr_target_alignment): New. (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. (set_dr_target_alignment): New. (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. * tree-vect-data-refs.c (dr_misalignment): Compute and return the group members misalignment. (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. (vect_analyze_data_refs_alignment): Compute alignment only for the first element of a DR group. (vect_slp_analyze_node_alignment): Likewise. --- gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- gcc/tree-vectorizer.h | 24 ++++++++++----- 2 files changed, 57 insertions(+), 32 deletions(-)
Comments
Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > This changes us to maintain and compute (mis-)alignment info for > the first element of a group only rather than for each DR when > doing interleaving and for the earliest, first, or first in the SLP > node (or any pair or all three of those) when SLP vectorizing. > > For this to work out the easiest way I have changed the accessors > DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to > the first element rather than adjusting all callers. > dr_misalignment is moved out-of-line and I'm not too fond of the > poly-int dances there (any hints?), but basically we are now > adjusting the first elements misalignment based on the DR_INIT > difference. > > Bootstrap & regtest running on x86_64-unknown-linux-gnu. > > Richard. > > 2021-09-13 Richard Biener <rguenther@suse.de> > > * tree-vectorizer.h (dr_misalignment): Move out of line. > (dr_target_alignment): New. > (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. > (set_dr_target_alignment): New. > (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. > * tree-vect-data-refs.c (dr_misalignment): Compute and > return the group members misalignment. > (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. > (vect_analyze_data_refs_alignment): Compute alignment only > for the first element of a DR group. > (vect_slp_analyze_node_alignment): Likewise. > --- > gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- > gcc/tree-vectorizer.h | 24 ++++++++++----- > 2 files changed, 57 insertions(+), 32 deletions(-) > > diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c > index 66e76132d14..b53d6a0b3f1 100644 > --- a/gcc/tree-vect-data-refs.c > +++ b/gcc/tree-vect-data-refs.c > @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) > return res; > } > > +/* Return the misalignment of DR_INFO. */ > + > +int > +dr_misalignment (dr_vec_info *dr_info) > +{ > + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) > + { > + dr_vec_info *first_dr > + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); > + int misalign = first_dr->misalignment; > + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); > + if (misalign == DR_MISALIGNMENT_UNKNOWN) > + return misalign; > + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) > + - wi::to_poly_offset (DR_INIT (first_dr->dr))); > + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); > + bool res = known_misalignment (mispoly, > + first_dr->target_alignment.to_constant (), > + &misalign); > + gcc_assert (res); > + return misalign; Yeah, not too keen on the to_constants here. The one on diff looks redundant -- you could just use diff.force_shwi () instead, and keep everything poly_int. For the known_misalignment I think we should use: if (!can_div_trunc_p (mispoly, first_dr->target_alignment, "ient, &misalign)) misalign = DR_MISALIGNMENT_UNKNOWN; return misalign; There are then no to_constant assumptions. Thanks, Richard
On Tue, 14 Sep 2021, Richard Sandiford wrote: > Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > > This changes us to maintain and compute (mis-)alignment info for > > the first element of a group only rather than for each DR when > > doing interleaving and for the earliest, first, or first in the SLP > > node (or any pair or all three of those) when SLP vectorizing. > > > > For this to work out the easiest way I have changed the accessors > > DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to > > the first element rather than adjusting all callers. > > dr_misalignment is moved out-of-line and I'm not too fond of the > > poly-int dances there (any hints?), but basically we are now > > adjusting the first elements misalignment based on the DR_INIT > > difference. > > > > Bootstrap & regtest running on x86_64-unknown-linux-gnu. > > > > Richard. > > > > 2021-09-13 Richard Biener <rguenther@suse.de> > > > > * tree-vectorizer.h (dr_misalignment): Move out of line. > > (dr_target_alignment): New. > > (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. > > (set_dr_target_alignment): New. > > (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. > > * tree-vect-data-refs.c (dr_misalignment): Compute and > > return the group members misalignment. > > (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. > > (vect_analyze_data_refs_alignment): Compute alignment only > > for the first element of a DR group. > > (vect_slp_analyze_node_alignment): Likewise. > > --- > > gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- > > gcc/tree-vectorizer.h | 24 ++++++++++----- > > 2 files changed, 57 insertions(+), 32 deletions(-) > > > > diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c > > index 66e76132d14..b53d6a0b3f1 100644 > > --- a/gcc/tree-vect-data-refs.c > > +++ b/gcc/tree-vect-data-refs.c > > @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) > > return res; > > } > > > > +/* Return the misalignment of DR_INFO. */ > > + > > +int > > +dr_misalignment (dr_vec_info *dr_info) > > +{ > > + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) > > + { > > + dr_vec_info *first_dr > > + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); > > + int misalign = first_dr->misalignment; > > + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); > > + if (misalign == DR_MISALIGNMENT_UNKNOWN) > > + return misalign; > > + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) > > + - wi::to_poly_offset (DR_INIT (first_dr->dr))); > > + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); > > + bool res = known_misalignment (mispoly, > > + first_dr->target_alignment.to_constant (), > > + &misalign); > > + gcc_assert (res); > > + return misalign; > > Yeah, not too keen on the to_constants here. The one on diff looks > redundant -- you could just use diff.force_shwi () instead, and > keep everything poly_int. > > For the known_misalignment I think we should use: > > if (!can_div_trunc_p (mispoly, first_dr->target_alignment, > "ient, &misalign)) > misalign = DR_MISALIGNMENT_UNKNOWN; > return misalign; > > There are then no to_constant assumptions. OK, note that group analysis does /* Check that the DR_INITs are compile-time constants. */ if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST || TREE_CODE (DR_INIT (drb)) != INTEGER_CST) break; /* Sorting has ensured that DR_INIT (dra) <= DR_INIT (drb). */ HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_INIT (dra)); HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_INIT (drb)); so I'm confident my variant was "correct", but it still was ugly. There's also the issue that target_alignment is poly_uint64 but misalignment is signed int. Note that can_div_trunc_p seems to require a poly_uint64 remainder, I'm not sure what to do with that, so I used is_constant. Btw, to what value do we want to align with variable sized vectors? We are aligning globals according to target_alignment but only support that for constant alignment. Most users only want to distinguish between aligned or not aligned and the actual misalignment is only used to seed the SSA alignment info, so I suppose being too conservative in dr_misalignment for variable size vectors isn't too bad if we correctly identify fully aligned accesses? I'm now testing a variant that looks like int dr_misalignment (dr_vec_info *dr_info) { if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) { dr_vec_info *first_dr = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); int misalign = first_dr->misalignment; gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); if (misalign == DR_MISALIGNMENT_UNKNOWN) return misalign; /* vect_analyze_data_ref_accesses guarantees that DR_INIT are INTEGER_CSTs and the first element in the group has the lowest address. */ poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) - wi::to_poly_offset (DR_INIT (first_dr->dr))); gcc_assert (known_ge (diff, 0)); poly_uint64 mispoly = misalign + diff.force_uhwi (); unsigned HOST_WIDE_INT quotient; if (can_div_trunc_p (mispoly, first_dr->target_alignment, "ient, &mispoly) && mispoly.is_constant ()) return mispoly.to_constant (); return DR_MISALIGNMENT_UNKNOWN; I wonder why a non-poly works for the quotient here. I suppose it's because the runtime factor is the same for all poly-ints and thus it cancels out? But then the remainder likely will never be constant unless it is zero? Thanks, Richard.
Richard Biener <rguenther@suse.de> writes: > On Tue, 14 Sep 2021, Richard Sandiford wrote: > >> Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: >> > This changes us to maintain and compute (mis-)alignment info for >> > the first element of a group only rather than for each DR when >> > doing interleaving and for the earliest, first, or first in the SLP >> > node (or any pair or all three of those) when SLP vectorizing. >> > >> > For this to work out the easiest way I have changed the accessors >> > DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to >> > the first element rather than adjusting all callers. >> > dr_misalignment is moved out-of-line and I'm not too fond of the >> > poly-int dances there (any hints?), but basically we are now >> > adjusting the first elements misalignment based on the DR_INIT >> > difference. >> > >> > Bootstrap & regtest running on x86_64-unknown-linux-gnu. >> > >> > Richard. >> > >> > 2021-09-13 Richard Biener <rguenther@suse.de> >> > >> > * tree-vectorizer.h (dr_misalignment): Move out of line. >> > (dr_target_alignment): New. >> > (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. >> > (set_dr_target_alignment): New. >> > (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. >> > * tree-vect-data-refs.c (dr_misalignment): Compute and >> > return the group members misalignment. >> > (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. >> > (vect_analyze_data_refs_alignment): Compute alignment only >> > for the first element of a DR group. >> > (vect_slp_analyze_node_alignment): Likewise. >> > --- >> > gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- >> > gcc/tree-vectorizer.h | 24 ++++++++++----- >> > 2 files changed, 57 insertions(+), 32 deletions(-) >> > >> > diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c >> > index 66e76132d14..b53d6a0b3f1 100644 >> > --- a/gcc/tree-vect-data-refs.c >> > +++ b/gcc/tree-vect-data-refs.c >> > @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) >> > return res; >> > } >> > >> > +/* Return the misalignment of DR_INFO. */ >> > + >> > +int >> > +dr_misalignment (dr_vec_info *dr_info) >> > +{ >> > + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) >> > + { >> > + dr_vec_info *first_dr >> > + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); >> > + int misalign = first_dr->misalignment; >> > + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); >> > + if (misalign == DR_MISALIGNMENT_UNKNOWN) >> > + return misalign; >> > + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) >> > + - wi::to_poly_offset (DR_INIT (first_dr->dr))); >> > + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); >> > + bool res = known_misalignment (mispoly, >> > + first_dr->target_alignment.to_constant (), >> > + &misalign); >> > + gcc_assert (res); >> > + return misalign; >> >> Yeah, not too keen on the to_constants here. The one on diff looks >> redundant -- you could just use diff.force_shwi () instead, and >> keep everything poly_int. >> >> For the known_misalignment I think we should use: >> >> if (!can_div_trunc_p (mispoly, first_dr->target_alignment, >> "ient, &misalign)) >> misalign = DR_MISALIGNMENT_UNKNOWN; >> return misalign; >> >> There are then no to_constant assumptions. > > OK, note that group analysis does > > /* Check that the DR_INITs are compile-time constants. */ > if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST > || TREE_CODE (DR_INIT (drb)) != INTEGER_CST) > break; > > /* Sorting has ensured that DR_INIT (dra) <= DR_INIT (drb). */ > HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_INIT (dra)); > HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_INIT (drb)); > > so I'm confident my variant was "correct", but it still was ugly. Ah, OK. In that case I don't mind the original version, but it would be good to have a comment above the to_constant saying where the condition is enforced. I'm just trying to avoid to_constant calls with no comment to explain them, and with no nearby is_constant call. Otherwise it could end up a bit like tree_to_uhwi, where sometimes tree_fits_uhwi_p really has been checked earlier (not always obvious where) and sometimes tree_to_uhwi is just used out of hope, to avoid having to think about the alternative. > There's also the issue that target_alignment is poly_uint64 but > misalignment is signed int. > > Note that can_div_trunc_p seems to require a poly_uint64 remainder, > I'm not sure what to do with that, so I used is_constant. Ah, yeah, forgot about that sorry. I guess in that case, using is_constant on first_dr->target_alignment and sticking with known_misalignment would make sense. > Btw, to what value do we want to align with variable sized vectors? > We are aligning globals according to target_alignment but only > support that for constant alignment. Most users only want to > distinguish between aligned or not aligned and the actual misalignment > is only used to seed the SSA alignment info, so I suppose being > too conservative in dr_misalignment for variable size vectors isn't > too bad if we correctly identify fully aligned accesses? In practice we don't require more than element alignment for VLA SVE as things stand. However, there's support for using fully-predicated loops in which the first loop iteration aligns by using leading inactive lanes where possible (e.g. start with 1 inactive lane for a misalignment of 1). In principle that would work for VLA too, we just haven't implemented it yet. > I'm now testing a variant that looks like > > int > dr_misalignment (dr_vec_info *dr_info) > { > if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) > { > dr_vec_info *first_dr > = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); > int misalign = first_dr->misalignment; > gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); > if (misalign == DR_MISALIGNMENT_UNKNOWN) > return misalign; > /* vect_analyze_data_ref_accesses guarantees that DR_INIT are > INTEGER_CSTs and the first element in the group has the lowest > address. */ > poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) > - wi::to_poly_offset (DR_INIT (first_dr->dr))); > gcc_assert (known_ge (diff, 0)); > poly_uint64 mispoly = misalign + diff.force_uhwi (); > unsigned HOST_WIDE_INT quotient; > if (can_div_trunc_p (mispoly, first_dr->target_alignment, "ient, > &mispoly) > && mispoly.is_constant ()) > return mispoly.to_constant (); > return DR_MISALIGNMENT_UNKNOWN; > > I wonder why a non-poly works for the quotient here. I suppose it's > because the runtime factor is the same for all poly-ints and thus > it cancels out? But then the remainder likely will never be constant > unless it is zero? It's not so much that the quotient is guaranteed to be constant, but that that particular overload is designed for the case in which the caller only cares about constant quotients, and wants can_div_trunc_p to fail otherwise. In other words, it's really just a way of cutting down on is_constant calls. poly-int.h doesn't provide every possible overload of can_div_trunc_p. In principle it could have a fully-general 4-poly_int version, but we haven't needed that yet. It could also have a new overload for the case we care about here, avoiding the separate is_constant. Thanks, Richard
On Wed, 15 Sep 2021, Richard Sandiford wrote: > Richard Biener <rguenther@suse.de> writes: > > On Tue, 14 Sep 2021, Richard Sandiford wrote: > > > >> Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > >> > This changes us to maintain and compute (mis-)alignment info for > >> > the first element of a group only rather than for each DR when > >> > doing interleaving and for the earliest, first, or first in the SLP > >> > node (or any pair or all three of those) when SLP vectorizing. > >> > > >> > For this to work out the easiest way I have changed the accessors > >> > DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to > >> > the first element rather than adjusting all callers. > >> > dr_misalignment is moved out-of-line and I'm not too fond of the > >> > poly-int dances there (any hints?), but basically we are now > >> > adjusting the first elements misalignment based on the DR_INIT > >> > difference. > >> > > >> > Bootstrap & regtest running on x86_64-unknown-linux-gnu. > >> > > >> > Richard. > >> > > >> > 2021-09-13 Richard Biener <rguenther@suse.de> > >> > > >> > * tree-vectorizer.h (dr_misalignment): Move out of line. > >> > (dr_target_alignment): New. > >> > (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. > >> > (set_dr_target_alignment): New. > >> > (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. > >> > * tree-vect-data-refs.c (dr_misalignment): Compute and > >> > return the group members misalignment. > >> > (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. > >> > (vect_analyze_data_refs_alignment): Compute alignment only > >> > for the first element of a DR group. > >> > (vect_slp_analyze_node_alignment): Likewise. > >> > --- > >> > gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- > >> > gcc/tree-vectorizer.h | 24 ++++++++++----- > >> > 2 files changed, 57 insertions(+), 32 deletions(-) > >> > > >> > diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c > >> > index 66e76132d14..b53d6a0b3f1 100644 > >> > --- a/gcc/tree-vect-data-refs.c > >> > +++ b/gcc/tree-vect-data-refs.c > >> > @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) > >> > return res; > >> > } > >> > > >> > +/* Return the misalignment of DR_INFO. */ > >> > + > >> > +int > >> > +dr_misalignment (dr_vec_info *dr_info) > >> > +{ > >> > + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) > >> > + { > >> > + dr_vec_info *first_dr > >> > + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); > >> > + int misalign = first_dr->misalignment; > >> > + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); > >> > + if (misalign == DR_MISALIGNMENT_UNKNOWN) > >> > + return misalign; > >> > + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) > >> > + - wi::to_poly_offset (DR_INIT (first_dr->dr))); > >> > + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); > >> > + bool res = known_misalignment (mispoly, > >> > + first_dr->target_alignment.to_constant (), > >> > + &misalign); > >> > + gcc_assert (res); > >> > + return misalign; > >> > >> Yeah, not too keen on the to_constants here. The one on diff looks > >> redundant -- you could just use diff.force_shwi () instead, and > >> keep everything poly_int. > >> > >> For the known_misalignment I think we should use: > >> > >> if (!can_div_trunc_p (mispoly, first_dr->target_alignment, > >> "ient, &misalign)) > >> misalign = DR_MISALIGNMENT_UNKNOWN; > >> return misalign; > >> > >> There are then no to_constant assumptions. > > > > OK, note that group analysis does > > > > /* Check that the DR_INITs are compile-time constants. */ > > if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST > > || TREE_CODE (DR_INIT (drb)) != INTEGER_CST) > > break; > > > > /* Sorting has ensured that DR_INIT (dra) <= DR_INIT (drb). */ > > HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_INIT (dra)); > > HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_INIT (drb)); > > > > so I'm confident my variant was "correct", but it still was ugly. > > Ah, OK. In that case I don't mind the original version, but it would be > good to have a comment above the to_constant saying where the condition > is enforced. > > I'm just trying to avoid to_constant calls with no comment to explain > them, and with no nearby is_constant call. Otherwise it could end up > a bit like tree_to_uhwi, where sometimes tree_fits_uhwi_p really has > been checked earlier (not always obvious where) and sometimes > tree_to_uhwi is just used out of hope, to avoid having to think about > the alternative. > > > There's also the issue that target_alignment is poly_uint64 but > > misalignment is signed int. > > > > Note that can_div_trunc_p seems to require a poly_uint64 remainder, > > I'm not sure what to do with that, so I used is_constant. > > Ah, yeah, forgot about that sorry. I guess in that case, using > is_constant on first_dr->target_alignment and sticking with > known_misalignment would make sense. > > > Btw, to what value do we want to align with variable sized vectors? > > We are aligning globals according to target_alignment but only > > support that for constant alignment. Most users only want to > > distinguish between aligned or not aligned and the actual misalignment > > is only used to seed the SSA alignment info, so I suppose being > > too conservative in dr_misalignment for variable size vectors isn't > > too bad if we correctly identify fully aligned accesses? > > In practice we don't require more than element alignment for VLA SVE > as things stand. However, there's support for using fully-predicated > loops in which the first loop iteration aligns by using leading > inactive lanes where possible (e.g. start with 1 inactive lane > for a misalignment of 1). In principle that would work for VLA too, > we just haven't implemented it yet. > > > I'm now testing a variant that looks like > > > > int > > dr_misalignment (dr_vec_info *dr_info) > > { > > if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) > > { > > dr_vec_info *first_dr > > = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); > > int misalign = first_dr->misalignment; > > gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); > > if (misalign == DR_MISALIGNMENT_UNKNOWN) > > return misalign; > > /* vect_analyze_data_ref_accesses guarantees that DR_INIT are > > INTEGER_CSTs and the first element in the group has the lowest > > address. */ > > poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) > > - wi::to_poly_offset (DR_INIT (first_dr->dr))); > > gcc_assert (known_ge (diff, 0)); > > poly_uint64 mispoly = misalign + diff.force_uhwi (); > > unsigned HOST_WIDE_INT quotient; > > if (can_div_trunc_p (mispoly, first_dr->target_alignment, "ient, > > &mispoly) > > && mispoly.is_constant ()) > > return mispoly.to_constant (); > > return DR_MISALIGNMENT_UNKNOWN; > > > > I wonder why a non-poly works for the quotient here. I suppose it's > > because the runtime factor is the same for all poly-ints and thus > > it cancels out? But then the remainder likely will never be constant > > unless it is zero? > > It's not so much that the quotient is guaranteed to be constant, > but that that particular overload is designed for the case in which > the caller only cares about constant quotients, and wants can_div_trunc_p > to fail otherwise. In other words, it's really just a way of cutting down > on is_constant calls. > > poly-int.h doesn't provide every possible overload of can_div_trunc_p. > In principle it could have a fully-general 4-poly_int version, but we > haven't needed that yet. It could also have a new overload for the > case we care about here, avoiding the separate is_constant. Hmm. I guess I'll see to somehow unify this with the logic in vect_compute_data_ref_alignment which at the moment forces DR_MISALIGNMENT to unknown if DR_TARGET_ALIGNMENT of the vector type isn't constant (so for SVE all accesses appear unaligned?). In the end vect_compute_data_ref_alignment does poly_int64 misalignment = base_misalignment + wi::to_poly_offset (drb->init).force_shwi (); /* If this is a backward running DR then first access in the larger vectype actually is N-1 elements before the address in the DR. Adjust misalign accordingly. */ if (tree_int_cst_sgn (drb->step) < 0) /* PLUS because STEP is negative. */ misalignment += ((TYPE_VECTOR_SUBPARTS (vectype) - 1) * -TREE_INT_CST_LOW (TYPE_SIZE_UNIT (TREE_TYPE (vectype)))); unsigned int const_misalignment; if (!known_misalignment (misalignment, vect_align_c, &const_misalignment)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "Non-constant misalignment for access: %T\n", ref); return; } SET_DR_MISALIGNMENT (dr_info, const_misalignment); ignoring the negative step offset (where poly-int TYPE_VECTOR_SUBPARTS sneaks in) this combines integer base_misaligned with DR_INIT (known INTEGER_CST) and then with the known constant vect_align_c. So the most simplest version is if (misalign == DR_MISALIGNMENT_UNKNOWN) return misalign; /* vect_analyze_data_ref_accesses guarantees that DR_INIT are INTEGER_CSTs and the first element in the group has the lowest address. Likewise vect_compute_data_ref_alignment will have ensured that target_alignment is constant and otherwise set misalign to DR_MISALIGNMENT_UNKNOWN. */ HOST_WIDE_INT diff = (TREE_INT_CST_LOW (DR_INIT (dr_info->dr)) - TREE_INT_CST_LOW (DR_INIT (first_dr->dr))); gcc_assert (diff >= 0); unsigned HOST_WIDE_INT target_alignment_c = first_dr->target_alignment.to_constant (); return (misalign + diff) % target_alignment_c; Note my intent is to lift the restriction of having only one vector type for vectorizing a DR group and alignment is where currently correctness issues pop up so in the end dr_misalignment () would get a vectype parameter (and the odd negative STEP handling would move to the caller, eventually with another 'offset' parameter to dr_misalignment () / aligned_access_p ()). So if you are fine with improving the situation for poly-ints after all this then I'd go with the most simplest variant above. OK? Thanks, Richard.
Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: > On Wed, 15 Sep 2021, Richard Sandiford wrote: >> Richard Biener <rguenther@suse.de> writes: >> > On Tue, 14 Sep 2021, Richard Sandiford wrote: >> > >> >> Richard Biener via Gcc-patches <gcc-patches@gcc.gnu.org> writes: >> >> > This changes us to maintain and compute (mis-)alignment info for >> >> > the first element of a group only rather than for each DR when >> >> > doing interleaving and for the earliest, first, or first in the SLP >> >> > node (or any pair or all three of those) when SLP vectorizing. >> >> > >> >> > For this to work out the easiest way I have changed the accessors >> >> > DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to >> >> > the first element rather than adjusting all callers. >> >> > dr_misalignment is moved out-of-line and I'm not too fond of the >> >> > poly-int dances there (any hints?), but basically we are now >> >> > adjusting the first elements misalignment based on the DR_INIT >> >> > difference. >> >> > >> >> > Bootstrap & regtest running on x86_64-unknown-linux-gnu. >> >> > >> >> > Richard. >> >> > >> >> > 2021-09-13 Richard Biener <rguenther@suse.de> >> >> > >> >> > * tree-vectorizer.h (dr_misalignment): Move out of line. >> >> > (dr_target_alignment): New. >> >> > (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. >> >> > (set_dr_target_alignment): New. >> >> > (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. >> >> > * tree-vect-data-refs.c (dr_misalignment): Compute and >> >> > return the group members misalignment. >> >> > (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. >> >> > (vect_analyze_data_refs_alignment): Compute alignment only >> >> > for the first element of a DR group. >> >> > (vect_slp_analyze_node_alignment): Likewise. >> >> > --- >> >> > gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- >> >> > gcc/tree-vectorizer.h | 24 ++++++++++----- >> >> > 2 files changed, 57 insertions(+), 32 deletions(-) >> >> > >> >> > diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c >> >> > index 66e76132d14..b53d6a0b3f1 100644 >> >> > --- a/gcc/tree-vect-data-refs.c >> >> > +++ b/gcc/tree-vect-data-refs.c >> >> > @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) >> >> > return res; >> >> > } >> >> > >> >> > +/* Return the misalignment of DR_INFO. */ >> >> > + >> >> > +int >> >> > +dr_misalignment (dr_vec_info *dr_info) >> >> > +{ >> >> > + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) >> >> > + { >> >> > + dr_vec_info *first_dr >> >> > + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); >> >> > + int misalign = first_dr->misalignment; >> >> > + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); >> >> > + if (misalign == DR_MISALIGNMENT_UNKNOWN) >> >> > + return misalign; >> >> > + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) >> >> > + - wi::to_poly_offset (DR_INIT (first_dr->dr))); >> >> > + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); >> >> > + bool res = known_misalignment (mispoly, >> >> > + first_dr->target_alignment.to_constant (), >> >> > + &misalign); >> >> > + gcc_assert (res); >> >> > + return misalign; >> >> >> >> Yeah, not too keen on the to_constants here. The one on diff looks >> >> redundant -- you could just use diff.force_shwi () instead, and >> >> keep everything poly_int. >> >> >> >> For the known_misalignment I think we should use: >> >> >> >> if (!can_div_trunc_p (mispoly, first_dr->target_alignment, >> >> "ient, &misalign)) >> >> misalign = DR_MISALIGNMENT_UNKNOWN; >> >> return misalign; >> >> >> >> There are then no to_constant assumptions. >> > >> > OK, note that group analysis does >> > >> > /* Check that the DR_INITs are compile-time constants. */ >> > if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST >> > || TREE_CODE (DR_INIT (drb)) != INTEGER_CST) >> > break; >> > >> > /* Sorting has ensured that DR_INIT (dra) <= DR_INIT (drb). */ >> > HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_INIT (dra)); >> > HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_INIT (drb)); >> > >> > so I'm confident my variant was "correct", but it still was ugly. >> >> Ah, OK. In that case I don't mind the original version, but it would be >> good to have a comment above the to_constant saying where the condition >> is enforced. >> >> I'm just trying to avoid to_constant calls with no comment to explain >> them, and with no nearby is_constant call. Otherwise it could end up >> a bit like tree_to_uhwi, where sometimes tree_fits_uhwi_p really has >> been checked earlier (not always obvious where) and sometimes >> tree_to_uhwi is just used out of hope, to avoid having to think about >> the alternative. >> >> > There's also the issue that target_alignment is poly_uint64 but >> > misalignment is signed int. >> > >> > Note that can_div_trunc_p seems to require a poly_uint64 remainder, >> > I'm not sure what to do with that, so I used is_constant. >> >> Ah, yeah, forgot about that sorry. I guess in that case, using >> is_constant on first_dr->target_alignment and sticking with >> known_misalignment would make sense. >> >> > Btw, to what value do we want to align with variable sized vectors? >> > We are aligning globals according to target_alignment but only >> > support that for constant alignment. Most users only want to >> > distinguish between aligned or not aligned and the actual misalignment >> > is only used to seed the SSA alignment info, so I suppose being >> > too conservative in dr_misalignment for variable size vectors isn't >> > too bad if we correctly identify fully aligned accesses? >> >> In practice we don't require more than element alignment for VLA SVE >> as things stand. However, there's support for using fully-predicated >> loops in which the first loop iteration aligns by using leading >> inactive lanes where possible (e.g. start with 1 inactive lane >> for a misalignment of 1). In principle that would work for VLA too, >> we just haven't implemented it yet. >> >> > I'm now testing a variant that looks like >> > >> > int >> > dr_misalignment (dr_vec_info *dr_info) >> > { >> > if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) >> > { >> > dr_vec_info *first_dr >> > = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); >> > int misalign = first_dr->misalignment; >> > gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); >> > if (misalign == DR_MISALIGNMENT_UNKNOWN) >> > return misalign; >> > /* vect_analyze_data_ref_accesses guarantees that DR_INIT are >> > INTEGER_CSTs and the first element in the group has the lowest >> > address. */ >> > poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) >> > - wi::to_poly_offset (DR_INIT (first_dr->dr))); >> > gcc_assert (known_ge (diff, 0)); >> > poly_uint64 mispoly = misalign + diff.force_uhwi (); >> > unsigned HOST_WIDE_INT quotient; >> > if (can_div_trunc_p (mispoly, first_dr->target_alignment, "ient, >> > &mispoly) >> > && mispoly.is_constant ()) >> > return mispoly.to_constant (); >> > return DR_MISALIGNMENT_UNKNOWN; >> > >> > I wonder why a non-poly works for the quotient here. I suppose it's >> > because the runtime factor is the same for all poly-ints and thus >> > it cancels out? But then the remainder likely will never be constant >> > unless it is zero? >> >> It's not so much that the quotient is guaranteed to be constant, >> but that that particular overload is designed for the case in which >> the caller only cares about constant quotients, and wants can_div_trunc_p >> to fail otherwise. In other words, it's really just a way of cutting down >> on is_constant calls. >> >> poly-int.h doesn't provide every possible overload of can_div_trunc_p. >> In principle it could have a fully-general 4-poly_int version, but we >> haven't needed that yet. It could also have a new overload for the >> case we care about here, avoiding the separate is_constant. > > Hmm. I guess I'll see to somehow unify this with the logic in > vect_compute_data_ref_alignment which at the moment forces > DR_MISALIGNMENT to unknown if DR_TARGET_ALIGNMENT of the vector type > isn't constant (so for SVE all accesses appear unaligned?). In the > end vect_compute_data_ref_alignment does > > poly_int64 misalignment > = base_misalignment + wi::to_poly_offset (drb->init).force_shwi (); > > /* If this is a backward running DR then first access in the larger > vectype actually is N-1 elements before the address in the DR. > Adjust misalign accordingly. */ > if (tree_int_cst_sgn (drb->step) < 0) > /* PLUS because STEP is negative. */ > misalignment += ((TYPE_VECTOR_SUBPARTS (vectype) - 1) > * -TREE_INT_CST_LOW (TYPE_SIZE_UNIT (TREE_TYPE > (vectype)))); > > unsigned int const_misalignment; > if (!known_misalignment (misalignment, vect_align_c, > &const_misalignment)) > { > if (dump_enabled_p ()) > dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, > "Non-constant misalignment for access: %T\n", > ref); > return; > } > > SET_DR_MISALIGNMENT (dr_info, const_misalignment); > > ignoring the negative step offset (where poly-int TYPE_VECTOR_SUBPARTS > sneaks in) this combines integer base_misaligned with DR_INIT > (known INTEGER_CST) and then with the known constant vect_align_c. > > So the most simplest version is > > if (misalign == DR_MISALIGNMENT_UNKNOWN) > return misalign; > /* vect_analyze_data_ref_accesses guarantees that DR_INIT are > INTEGER_CSTs and the first element in the group has the lowest > address. Likewise vect_compute_data_ref_alignment will > have ensured that target_alignment is constant and otherwise > set misalign to DR_MISALIGNMENT_UNKNOWN. */ > HOST_WIDE_INT diff = (TREE_INT_CST_LOW (DR_INIT (dr_info->dr)) > - TREE_INT_CST_LOW (DR_INIT (first_dr->dr))); > gcc_assert (diff >= 0); > unsigned HOST_WIDE_INT target_alignment_c > = first_dr->target_alignment.to_constant (); > return (misalign + diff) % target_alignment_c; LGTM, thanks. Richard > Note my intent is to lift the restriction of having only one > vector type for vectorizing a DR group and alignment is where > currently correctness issues pop up so in the end > dr_misalignment () would get a vectype parameter (and the > odd negative STEP handling would move to the caller, eventually > with another 'offset' parameter to dr_misalignment () / aligned_access_p > ()). > > So if you are fine with improving the situation for poly-ints > after all this then I'd go with the most simplest variant above. > > OK? > > Thanks, > Richard.
diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c index 66e76132d14..b53d6a0b3f1 100644 --- a/gcc/tree-vect-data-refs.c +++ b/gcc/tree-vect-data-refs.c @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) return res; } +/* Return the misalignment of DR_INFO. */ + +int +dr_misalignment (dr_vec_info *dr_info) +{ + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) + { + dr_vec_info *first_dr + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); + int misalign = first_dr->misalignment; + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); + if (misalign == DR_MISALIGNMENT_UNKNOWN) + return misalign; + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) + - wi::to_poly_offset (DR_INIT (first_dr->dr))); + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); + bool res = known_misalignment (mispoly, + first_dr->target_alignment.to_constant (), + &misalign); + gcc_assert (res); + return misalign; + } + else + { + int misalign = dr_info->misalignment; + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); + return misalign; + } +} + /* Record the base alignment guarantee given by DRB, which occurs in STMT_INFO. */ @@ -992,7 +1022,7 @@ vect_compute_data_ref_alignment (vec_info *vinfo, dr_vec_info *dr_info) poly_uint64 vector_alignment = exact_div (vect_calculate_target_alignment (dr_info), BITS_PER_UNIT); - DR_TARGET_ALIGNMENT (dr_info) = vector_alignment; + SET_DR_TARGET_ALIGNMENT (dr_info, vector_alignment); /* If the main loop has peeled for alignment we have no way of knowing whether the data accesses in the epilogues are aligned. We can't at @@ -2408,7 +2438,12 @@ vect_analyze_data_refs_alignment (loop_vec_info loop_vinfo) { dr_vec_info *dr_info = loop_vinfo->lookup_dr (dr); if (STMT_VINFO_VECTORIZABLE (dr_info->stmt)) - vect_compute_data_ref_alignment (loop_vinfo, dr_info); + { + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt) + && DR_GROUP_FIRST_ELEMENT (dr_info->stmt) != dr_info->stmt) + continue; + vect_compute_data_ref_alignment (loop_vinfo, dr_info); + } } return opt_result::success (); @@ -2420,13 +2455,9 @@ vect_analyze_data_refs_alignment (loop_vec_info loop_vinfo) static bool vect_slp_analyze_node_alignment (vec_info *vinfo, slp_tree node) { - /* We vectorize from the first scalar stmt in the node unless - the node is permuted in which case we start from the first - element in the group. */ + /* Alignment is maintained in the first element of the group. */ stmt_vec_info first_stmt_info = SLP_TREE_SCALAR_STMTS (node)[0]; - dr_vec_info *first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - if (SLP_TREE_LOAD_PERMUTATION (node).exists ()) - first_stmt_info = DR_GROUP_FIRST_ELEMENT (first_stmt_info); + first_stmt_info = DR_GROUP_FIRST_ELEMENT (first_stmt_info); /* We need to commit to a vector type for the group now. */ if (is_a <bb_vec_info> (vinfo) @@ -2440,22 +2471,8 @@ vect_slp_analyze_node_alignment (vec_info *vinfo, slp_tree node) } dr_vec_info *dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - vect_compute_data_ref_alignment (vinfo, dr_info); - /* In several places we need alignment of the first element anyway. */ - if (dr_info != first_dr_info) - vect_compute_data_ref_alignment (vinfo, first_dr_info); - - /* For creating the data-ref pointer we need alignment of the - first element as well. */ - first_stmt_info - = vect_stmt_to_vectorize (vect_find_first_scalar_stmt_in_slp (node)); - if (first_stmt_info != SLP_TREE_SCALAR_STMTS (node)[0]) - { - first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - if (dr_info != first_dr_info) - vect_compute_data_ref_alignment (vinfo, first_dr_info); - } - + if (dr_info->misalignment == DR_MISALIGNMENT_UNINITIALIZED) + vect_compute_data_ref_alignment (vinfo, dr_info); return true; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 91206eba1aa..8520bc0b6e5 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1606,13 +1606,7 @@ set_dr_misalignment (dr_vec_info *dr_info, int val) dr_info->misalignment = val; } -inline int -dr_misalignment (dr_vec_info *dr_info) -{ - int misalign = dr_info->misalignment; - gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); - return misalign; -} +extern int dr_misalignment (dr_vec_info *dr_info); /* Reflects actual alignment of first access in the vectorized loop, taking into account peeling/versioning if applied. */ @@ -1620,7 +1614,21 @@ dr_misalignment (dr_vec_info *dr_info) #define SET_DR_MISALIGNMENT(DR, VAL) set_dr_misalignment (DR, VAL) /* Only defined once DR_MISALIGNMENT is defined. */ -#define DR_TARGET_ALIGNMENT(DR) ((DR)->target_alignment) +static inline const poly_uint64 +dr_target_alignment (dr_vec_info *dr_info) +{ + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) + dr_info = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); + return dr_info->target_alignment; +} +#define DR_TARGET_ALIGNMENT(DR) dr_target_alignment (DR) + +static inline void +set_dr_target_alignment (dr_vec_info *dr_info, poly_uint64 val) +{ + dr_info->target_alignment = val; +} +#define SET_DR_TARGET_ALIGNMENT(DR, VAL) set_dr_target_alignment (DR, VAL) /* Return true if data access DR_INFO is aligned to its target alignment (which may be less than a full vector). */