From patchwork Mon Sep 13 14:37:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 44949 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0D2A73857416 for ; Mon, 13 Sep 2021 14:38:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0D2A73857416 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1631543911; bh=JfCOPPxT/3/Hyl04q/TvbeNoXDitCbTC005Pghgc1JY=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=WiBeyFhLlJTK/vWIgimR3O5vutMN7mGmnG4OkAyUnxcsKhMVK3X4jPa+0bi9N8fmv DY2YTovwyZs9yiBvvXds0+A57czYS/OWoNWKC1KMSNvaW+kPFK+hIKZJ3nhnBX1EZb zXMRPULHAZZuBe9MfsuOUzNejv16gQMudkFTuEh4= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id A5E113857413 for ; Mon, 13 Sep 2021 14:37:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A5E113857413 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9AB271FFF3; Mon, 13 Sep 2021 14:37:44 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 75F8213AB9; Mon, 13 Sep 2021 14:37:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id AauWGzhiP2EGagAAMHmgww (envelope-from ); Mon, 13 Sep 2021 14:37:44 +0000 Date: Mon, 13 Sep 2021 16:37:43 +0200 (CEST) To: gcc-patches@gcc.gnu.org Subject: [PATCH] Maintain (mis-)alignment info in the first element of a group Message-ID: <94r9n910-qsp7-778q-5ooq-pno71s1sqn6o@fhfr.qr> MIME-Version: 1.0 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Biener via Gcc-patches From: Richard Biener Reply-To: Richard Biener Cc: richard.sandiford@arm.com Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This changes us to maintain and compute (mis-)alignment info for the first element of a group only rather than for each DR when doing interleaving and for the earliest, first, or first in the SLP node (or any pair or all three of those) when SLP vectorizing. For this to work out the easiest way I have changed the accessors DR_MISALIGNMENT and DR_TARGET_ALIGNMENT to do the indirection to the first element rather than adjusting all callers. dr_misalignment is moved out-of-line and I'm not too fond of the poly-int dances there (any hints?), but basically we are now adjusting the first elements misalignment based on the DR_INIT difference. Bootstrap & regtest running on x86_64-unknown-linux-gnu. Richard. 2021-09-13 Richard Biener * tree-vectorizer.h (dr_misalignment): Move out of line. (dr_target_alignment): New. (DR_TARGET_ALIGNMENT): Wrap dr_target_alignment. (set_dr_target_alignment): New. (SET_DR_TARGET_ALIGNMENT): Wrap set_dr_target_alignment. * tree-vect-data-refs.c (dr_misalignment): Compute and return the group members misalignment. (vect_compute_data_ref_alignment): Use SET_DR_TARGET_ALIGNMENT. (vect_analyze_data_refs_alignment): Compute alignment only for the first element of a DR group. (vect_slp_analyze_node_alignment): Likewise. --- gcc/tree-vect-data-refs.c | 65 ++++++++++++++++++++++++--------------- gcc/tree-vectorizer.h | 24 ++++++++++----- 2 files changed, 57 insertions(+), 32 deletions(-) diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c index 66e76132d14..b53d6a0b3f1 100644 --- a/gcc/tree-vect-data-refs.c +++ b/gcc/tree-vect-data-refs.c @@ -887,6 +887,36 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) return res; } +/* Return the misalignment of DR_INFO. */ + +int +dr_misalignment (dr_vec_info *dr_info) +{ + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) + { + dr_vec_info *first_dr + = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); + int misalign = first_dr->misalignment; + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); + if (misalign == DR_MISALIGNMENT_UNKNOWN) + return misalign; + poly_offset_int diff = (wi::to_poly_offset (DR_INIT (dr_info->dr)) + - wi::to_poly_offset (DR_INIT (first_dr->dr))); + poly_int64 mispoly = misalign + diff.to_constant ().to_shwi (); + bool res = known_misalignment (mispoly, + first_dr->target_alignment.to_constant (), + &misalign); + gcc_assert (res); + return misalign; + } + else + { + int misalign = dr_info->misalignment; + gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); + return misalign; + } +} + /* Record the base alignment guarantee given by DRB, which occurs in STMT_INFO. */ @@ -992,7 +1022,7 @@ vect_compute_data_ref_alignment (vec_info *vinfo, dr_vec_info *dr_info) poly_uint64 vector_alignment = exact_div (vect_calculate_target_alignment (dr_info), BITS_PER_UNIT); - DR_TARGET_ALIGNMENT (dr_info) = vector_alignment; + SET_DR_TARGET_ALIGNMENT (dr_info, vector_alignment); /* If the main loop has peeled for alignment we have no way of knowing whether the data accesses in the epilogues are aligned. We can't at @@ -2408,7 +2438,12 @@ vect_analyze_data_refs_alignment (loop_vec_info loop_vinfo) { dr_vec_info *dr_info = loop_vinfo->lookup_dr (dr); if (STMT_VINFO_VECTORIZABLE (dr_info->stmt)) - vect_compute_data_ref_alignment (loop_vinfo, dr_info); + { + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt) + && DR_GROUP_FIRST_ELEMENT (dr_info->stmt) != dr_info->stmt) + continue; + vect_compute_data_ref_alignment (loop_vinfo, dr_info); + } } return opt_result::success (); @@ -2420,13 +2455,9 @@ vect_analyze_data_refs_alignment (loop_vec_info loop_vinfo) static bool vect_slp_analyze_node_alignment (vec_info *vinfo, slp_tree node) { - /* We vectorize from the first scalar stmt in the node unless - the node is permuted in which case we start from the first - element in the group. */ + /* Alignment is maintained in the first element of the group. */ stmt_vec_info first_stmt_info = SLP_TREE_SCALAR_STMTS (node)[0]; - dr_vec_info *first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - if (SLP_TREE_LOAD_PERMUTATION (node).exists ()) - first_stmt_info = DR_GROUP_FIRST_ELEMENT (first_stmt_info); + first_stmt_info = DR_GROUP_FIRST_ELEMENT (first_stmt_info); /* We need to commit to a vector type for the group now. */ if (is_a (vinfo) @@ -2440,22 +2471,8 @@ vect_slp_analyze_node_alignment (vec_info *vinfo, slp_tree node) } dr_vec_info *dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - vect_compute_data_ref_alignment (vinfo, dr_info); - /* In several places we need alignment of the first element anyway. */ - if (dr_info != first_dr_info) - vect_compute_data_ref_alignment (vinfo, first_dr_info); - - /* For creating the data-ref pointer we need alignment of the - first element as well. */ - first_stmt_info - = vect_stmt_to_vectorize (vect_find_first_scalar_stmt_in_slp (node)); - if (first_stmt_info != SLP_TREE_SCALAR_STMTS (node)[0]) - { - first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info); - if (dr_info != first_dr_info) - vect_compute_data_ref_alignment (vinfo, first_dr_info); - } - + if (dr_info->misalignment == DR_MISALIGNMENT_UNINITIALIZED) + vect_compute_data_ref_alignment (vinfo, dr_info); return true; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 91206eba1aa..8520bc0b6e5 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1606,13 +1606,7 @@ set_dr_misalignment (dr_vec_info *dr_info, int val) dr_info->misalignment = val; } -inline int -dr_misalignment (dr_vec_info *dr_info) -{ - int misalign = dr_info->misalignment; - gcc_assert (misalign != DR_MISALIGNMENT_UNINITIALIZED); - return misalign; -} +extern int dr_misalignment (dr_vec_info *dr_info); /* Reflects actual alignment of first access in the vectorized loop, taking into account peeling/versioning if applied. */ @@ -1620,7 +1614,21 @@ dr_misalignment (dr_vec_info *dr_info) #define SET_DR_MISALIGNMENT(DR, VAL) set_dr_misalignment (DR, VAL) /* Only defined once DR_MISALIGNMENT is defined. */ -#define DR_TARGET_ALIGNMENT(DR) ((DR)->target_alignment) +static inline const poly_uint64 +dr_target_alignment (dr_vec_info *dr_info) +{ + if (STMT_VINFO_GROUPED_ACCESS (dr_info->stmt)) + dr_info = STMT_VINFO_DR_INFO (DR_GROUP_FIRST_ELEMENT (dr_info->stmt)); + return dr_info->target_alignment; +} +#define DR_TARGET_ALIGNMENT(DR) dr_target_alignment (DR) + +static inline void +set_dr_target_alignment (dr_vec_info *dr_info, poly_uint64 val) +{ + dr_info->target_alignment = val; +} +#define SET_DR_TARGET_ALIGNMENT(DR, VAL) set_dr_target_alignment (DR, VAL) /* Return true if data access DR_INFO is aligned to its target alignment (which may be less than a full vector). */