From patchwork Tue Jan 16 14:43:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 84185 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C27C13861839 for ; Tue, 16 Jan 2024 14:48:59 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by sourceware.org (Postfix) with ESMTPS id 2995F385773D for ; Tue, 16 Jan 2024 14:48:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2995F385773D Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2995F385773D Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705416514; cv=none; b=wOQfj+b5pVEP2wkdjfiw5A5WuOySjtUO6wuynlt/QCA/kjvWu1VqxfTxICrFC2R10VGAR9ZwRAsE6lZxXJP5jNFrezqF7uEXmjIhb2b/U23QioF1KeOzbfAv8PeLAT8WthJurU07pIhm1Zw5ikCmWhz80gSu5jMuIUWx4PX62G8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705416514; c=relaxed/simple; bh=Rl/ulHEVA9H3uRDcoQVfLOZo446TIuamPqNp0xJba4w=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=f3kOuPH6cblZ0ln2TXDxT5lLhT7QCSTt7fpO4GKIJKTtvxwCemMTWRI9ndDoS+qEZRduMEyHtNCYNNfMEV7R4eBkRwMDCa66FZlM/BA3aKRugoFwwcEmiCf7LhdNNR8+e2768ViJosuxZrXibjttz1elxSfZo+sMjQkCRvltRcM= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from [10.168.4.150] (unknown [10.168.4.150]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C1FC81FBAF; Tue, 16 Jan 2024 14:48:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705416502; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=ROk3r/iU6EPpAQ2b3RSgjcrOrGV34W++PWWFpLuicIoKoemodu//RBxBXh+c8Ttzy9Z6mW x5eP0Bve5XPVkPxYorr14tlubJiadV5AheRbBSE10aCwQKHMd3Zpzk62vMtH63uY4VXzpG efE9div5AhI4YrXdTyOUuq80qyOJhHI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705416502; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=xqN1MQLKZotOQHSmVborf5uh0ecY/4q23ZlqCi9JgeyCE3SUsYveMJfg1KyAmMLNxSiYoD sUKqsPaoBkxuikAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705416501; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=gMZI7T0RX27VxbRZ8GWzxEV1LcLPIsdU7Noneenyi6lX1LK5Hr19jV8pEvJe1fR93gL+Im POGGRwjFyhuLURwRQGpUgo8UrfF2zoC6QCse8q+ot7dcAFD7DHWZeHp7M9PebdrW3+QxqH qq2XnGneQ59KdqIU4kFIWFTBxQj9054= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705416501; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=XhxaszfljLOtHW1WUublgEGzLy8aobdk6O1+WoqqFvcqNBfbXgPmfBMXEgxNEe8bY9Rls3 Fm62ocA9afl68tAw== Date: Tue, 16 Jan 2024 15:43:15 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org cc: tamar.christina@arm.com Subject: [PATCH] tree-optimization/113373 - work around early exit vect missing LC PHI MIME-Version: 1.0 Authentication-Results: smtp-out2.suse.de; none X-Spamd-Result: default: False [-0.60 / 50.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; MISSING_MID(2.50)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[tree-vect-loop.cc:url]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; BAYES_HAM(-3.00)[100.00%] X-Spam-Level: X-Spam-Score: -0.60 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Message-Id: <20240116144859.C27C13861839@sourceware.org> The following makes reduction epilogue code generation handle missing LC PHIs for the reductions, in particular also avoid replacing results on one exit with epilogue result from another one (but not avoiding generation of dead epilogues yet). The proper fix should be eventually to create the missing LC PHIs of course. Bootstrapped and tested on x86_64-unknown-linux-gnu, waiting for the pre-checkin CI. PR tree-optimization/113373 * tree-vect-loop.cc (vect_create_epilog_for_reduction): Only replace LC PHI results. Handle merge PHIs as seen for early breaks which miss the LC PHIs for the reduction results. * gcc.dg/vect/vect-early-break_104-pr113373.c: New testcase. --- .../vect/vect-early-break_104-pr113373.c | 19 ++++++ gcc/tree-vect-loop.cc | 67 +++++++++++-------- 2 files changed, 59 insertions(+), 27 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c new file mode 100644 index 00000000000..1601aafb3e6 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c @@ -0,0 +1,19 @@ +/* { dg-do compile } */ +/* { dg-add-options vect_early_break } */ +/* { dg-require-effective-target vect_early_break } */ + +struct asCArray { + unsigned *array; + int length; +}; +unsigned asCReaderTranslateFunction(struct asCArray b, unsigned t) +{ + int size = 0; + for (unsigned num; num < t; num++) + { + if (num >= b.length) + __builtin_abort(); + size += b.array[num]; + } + return size; +} diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index fedecd620e9..1c1b59dc5db 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -6013,7 +6013,6 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, gimple_stmt_iterator exit_gsi; tree new_temp = NULL_TREE, new_name, new_scalar_dest; gimple *epilog_stmt = NULL; - gimple *exit_phi; tree bitsize; tree def; tree orig_name, scalar_result; @@ -6024,7 +6023,6 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, int j, i; vec &scalar_results = reduc_info->reduc_scalar_results; unsigned int group_size = 1, k; - auto_vec phis; /* SLP reduction without reduction chain, e.g., # a1 = phi # b1 = phi @@ -6937,44 +6935,59 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, use */ gcc_assert (live_out_stmts.size () == scalar_results.length ()); + auto_vec phis; for (k = 0; k < live_out_stmts.size (); k++) { stmt_vec_info scalar_stmt_info = vect_orig_stmt (live_out_stmts[k]); scalar_dest = gimple_get_lhs (scalar_stmt_info->stmt); - phis.create (3); /* Find the loop-closed-use at the loop exit of the original scalar result. (The reduction result is expected to have two immediate uses, one at the latch block, and one at the loop exit). For double - reductions we are looking for exit phis of the outer loop. */ - FOR_EACH_IMM_USE_FAST (use_p, imm_iter, scalar_dest) - { - if (!flow_bb_inside_loop_p (loop, gimple_bb (USE_STMT (use_p)))) + reductions we are looking for exit phis of the outer loop. + ??? For early-breaks we mess up scalar LC SSA form so also match + a PHI in a merge block and replace that PHIs use on the correct + edge. */ + FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, scalar_dest) + { + if (is_gimple_debug (use_stmt)) + continue; + else if (!flow_bb_inside_loop_p (loop, gimple_bb (use_stmt))) { - if (!is_gimple_debug (USE_STMT (use_p))) - phis.safe_push (USE_STMT (use_p)); + if (gimple_bb (use_stmt) == loop_exit->dest) + phis.safe_push (as_a (use_stmt)); + else if (gimple_code (use_stmt) == GIMPLE_PHI) + { + /* ??? This is the case running into a merge PHI with + a missing LC PHI for early exit vectorization. Replace + only the merge argument corresponding to loop_exit. */ + FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter) + if (gimple_phi_arg_edge (as_a (use_stmt), + phi_arg_index_from_use (use_p)) + ->src == loop_exit->dest) + SET_USE (use_p, scalar_results[k]); + update_stmt (use_stmt); + } } - else - { - if (double_reduc && gimple_code (USE_STMT (use_p)) == GIMPLE_PHI) - { - tree phi_res = PHI_RESULT (USE_STMT (use_p)); + else if (double_reduc && gimple_code (use_stmt) == GIMPLE_PHI) + { + tree phi_res = PHI_RESULT (use_stmt); - FOR_EACH_IMM_USE_FAST (phi_use_p, phi_imm_iter, phi_res) - { - if (!flow_bb_inside_loop_p (loop, - gimple_bb (USE_STMT (phi_use_p))) - && !is_gimple_debug (USE_STMT (phi_use_p))) - phis.safe_push (USE_STMT (phi_use_p)); - } - } - } - } + FOR_EACH_IMM_USE_FAST (phi_use_p, phi_imm_iter, phi_res) + { + if (!flow_bb_inside_loop_p (outer_loop, + gimple_bb (USE_STMT (phi_use_p))) + && !is_gimple_debug (USE_STMT (phi_use_p))) + phis.safe_push (as_a (USE_STMT (phi_use_p))); + } + } + } + gphi *exit_phi; FOR_EACH_VEC_ELT (phis, i, exit_phi) { - /* Replace the uses: */ - orig_name = PHI_RESULT (exit_phi); + /* Replace the uses: */ + orig_name = gimple_phi_result (exit_phi); /* Look for a single use at the target of the skip edge. */ if (unify_with_main_loop_p) @@ -6995,7 +7008,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, } } - phis.release (); + phis.truncate (0); } }