From patchwork Tue Sep 1 13:08:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 142631 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 04E984BA7983 for ; Tue, 1 Sep 2026 13:09:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 04E984BA7983 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=XS193L+S; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=rOvml2hs; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=uqcD/PW4; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=Peawn86M X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by sourceware.org (Postfix) with ESMTPS id 3546D4BA2E0E for ; Tue, 1 Sep 2026 13:08:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3546D4BA2E0E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 3546D4BA2E0E Authentication-Results: sourceware.org; arc=none smtp.remote-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788268099; cv=none; b=I9aEuQNFr7ulRbM7INmgV5Bu/vUOW5P4zRE56LzWd47mbrOUrCn/SExmZAIAjqc4z0XLGBmEOTSFDCAqrJwCUenYz+UoFq6rZ92u1xMSitDO8YsoPLXzxthy5hji53HbwgfO+NfnjUXZda9o4/B4TjhaEafGr6lV643XbwJzouc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788268099; c=relaxed/simple; bh=I7/bIQdroUmjSF3sIbtyjhcn72yFAsH5arzRLN/nglY=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:Message-ID:MIME-Version; b=QkMX9qZsqgmIHkURRjRwKALYjC13sn/lXjGSmBom1gCpZpmkyF7x+VnsMW2vN1+6iAkBsug9oUnHMdUeDtrfc/EX3+usP8w9C/KNMAnzDvrZgLiVyFejmy2oXrGAboY9h+kLFRSW43A64dxoRwy8vbXmy4njC2Vd/7cHNH5iypw= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=XS193L+S; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=rOvml2hs; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=uqcD/PW4; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=Peawn86M DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3546D4BA2E0E Received: from murzim.nue2.suse.org (unknown [10.168.4.243]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 0A4FA223FA for ; Tue, 1 Sep 2026 13:08:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1788268094; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=aazJiPazYlO8zcS3vsstN6r2S9lCh2MwGCGyrEVt5/U=; b=XS193L+SkYFx2jpNHPfyaiMsIU2BGJ5jaHAOh1Rf5dmhBwG7ahZx2BtnjFbcOhQp2RVYxo RuOkNvEC5/s7N6gynBGX73+uN7HZEOFFtEmjvj4ylMQ5ICJ1cWA33dUR3OPb99rJJ4nncw T4unSWQvSgKP74afLr9waZbxbXbfMzY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1788268094; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=aazJiPazYlO8zcS3vsstN6r2S9lCh2MwGCGyrEVt5/U=; b=rOvml2hs7OtZ82HvfPC3NCxmkUEO9D3BtIS80oYBLsWtctiaxq6oWCIr/SxTE3rGmyvtzJ 2R4pelIttp04eZDg== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1788268090; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=aazJiPazYlO8zcS3vsstN6r2S9lCh2MwGCGyrEVt5/U=; b=uqcD/PW4rVMc7AgoANlEn62dofUldPhCEFl3XXVzPr6kUbaQ1zeucCDudMsnbyWbi8YAol EIEANHygdS88TPEqaQsbYBU2n81+CaOL1J30qL3tGQDt/L4NA6nY9BIRD89Gsx7i9uFAfc 8uBI54AY54e6d0+XSJJ6zBvxvfrkvKE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1788268090; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=aazJiPazYlO8zcS3vsstN6r2S9lCh2MwGCGyrEVt5/U=; b=Peawn86MiKkjrYVB9oyLqCC8wc2lIQle/83sx7F472iz9UuMQAIlbvu82q3Xwg9HbzQb6l PZKZ+sTchNU0W+Bg== Date: Tue, 1 Sep 2026 15:08:09 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] Use more precise costing location for BB vectorization Message-ID: <24o84ns1-9440-3q0r-q63r-rn6322o186oo@fhfr.qr> MIME-Version: 1.0 X-Spam-Score: -4.30 X-Spam-Level: X-Spamd-Result: default: False [-4.30 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-0.998]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ZERO(0.00)[0]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; FROM_HAS_DN(0.00)[] X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Since we are now computing a place to schedule SLP nodes, use that for costing. Deal with the fact that the vector part will have "outside" cost from vector constants and externs which get scheduled at region entry. Scalar costing has no such parts, so the logic that tries to assess all code regions are profitable to vectorize independently breaks apart because of the region containing the entry. Delay that region cost and assessed it only as part of the overall cost. Bootstrapped and tested on x86_64-unknown-linux-gnu. I'll see whether the CI will pick any fallout elsewhere. * tree-vect-slp.cc (vect_bb_vectorization_profitable_p): Prefer the computed SLP schedule location to determine the containing loop. Make sure to make lock-step progress when iterating regions with missing vector/scalar part. Delay individual assessment of profitability for the region containing the entry and require profitability based on total costs. --- gcc/tree-vect-slp.cc | 103 +++++++++++++++++++++++++++++-------------- 1 file changed, 69 insertions(+), 34 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 8564422028b..5fbdaedb58c 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -9837,57 +9837,71 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo, li_scalar_costs.quick_push (std::make_pair (l, cost)); } /* Use a random used loop as fallback in case the first vector_costs - entry does not have a stmt_info associated with it. */ + entry does not have a location associated with it. */ unsigned l = li_scalar_costs[0].first; FOR_EACH_VEC_ELT (vector_costs, i, cost) { - /* We inherit from the previous COST, invariants, externals and - extracts immediately follow the cost for the related stmt. */ - if (cost->stmt_info) + /* Use SLP node placement according to the computed schedule. */ + if (cost->node && cost->node->si) + l = gimple_bb (cost->node->si)->loop_father->num; + /* For schedules at region boundary use the region entry loop. */ + else if (cost->node) + l = bb_vinfo->bbs[0]->loop_father->num; + /* SLP instance root stmts do not have an associated SLP node. */ + else if (cost->stmt_info) l = gimple_bb (cost->stmt_info->stmt)->loop_father->num; + /* And since vect_prologue_cost_for_slp can end up costing with + neither, inherit from the previous node. */ li_vector_costs.quick_push (std::make_pair (l, cost)); } li_scalar_costs.stablesort (li_cost_vec_cmp, NULL); li_vector_costs.stablesort (li_cost_vec_cmp, NULL); + unsigned total_vec_outside_cost = 0; + unsigned total_vec_inside_cost = 0; + unsigned total_scalar_cost = 0; + /* Now cost the portions individually. */ unsigned vi = 0; unsigned si = 0; bool profitable = true; while (si < li_scalar_costs.length () - && vi < li_vector_costs.length ()) + || vi < li_vector_costs.length ()) { - unsigned sl = li_scalar_costs[si].first; - unsigned vl = li_vector_costs[vi].first; - if (sl != vl) + unsigned sl + = si < li_scalar_costs.length () ? li_scalar_costs[si].first : -1U; + unsigned vl + = vi < li_vector_costs.length () ? li_vector_costs[vi].first : -1U; + + class vector_costs *scalar_target_cost_data = nullptr; + scalar_cost = 0; + if (sl <= vl) { if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, - "Scalar %d and vector %d loop part do not " - "match up, skipping scalar part\n", sl, vl); - /* Skip the scalar part, assuming zero cost on the vector side. */ + "Scalar cost for part in loop %d\n", sl); + scalar_target_cost_data = init_cost (bb_vinfo, true); do { + add_stmt_cost (scalar_target_cost_data, + li_scalar_costs[si].second); si++; } while (si < li_scalar_costs.length () && li_scalar_costs[si].first == sl); - continue; - } - - if (dump_enabled_p ()) - dump_printf_loc (MSG_NOTE, vect_location, - "Scalar cost for part in loop %d\n", sl); - class vector_costs *scalar_target_cost_data = init_cost (bb_vinfo, true); - do - { - add_stmt_cost (scalar_target_cost_data, li_scalar_costs[si].second); - si++; + scalar_target_cost_data->finish_cost (nullptr); + scalar_cost = scalar_target_cost_data->body_cost (); + total_scalar_cost += scalar_cost; + if (sl < vl) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "Scalar %d loop part does not " + "have corresponding vector part\n", sl); + delete scalar_target_cost_data; + continue; + } } - while (si < li_scalar_costs.length () - && li_scalar_costs[si].first == sl); - scalar_target_cost_data->finish_cost (nullptr); - scalar_cost = scalar_target_cost_data->body_cost (); /* Complete the target-specific vector cost calculation. */ if (dump_enabled_p ()) @@ -9907,15 +9921,30 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo, vec_prologue_cost = vect_target_cost_data->prologue_cost (); vec_inside_cost = vect_target_cost_data->body_cost (); vec_epilogue_cost = vect_target_cost_data->epilogue_cost (); - delete scalar_target_cost_data; + if (scalar_target_cost_data) + delete scalar_target_cost_data; delete vect_target_cost_data; vec_outside_cost = vec_prologue_cost + vec_epilogue_cost; + total_vec_outside_cost += vec_outside_cost; + total_vec_inside_cost += vec_inside_cost; + + if (sl > vl && dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "Vector %d loop part does not " + "have corresponding scalar part\n", vl); + + /* When this is vector costs for the region entry delay costing + and instead only require the total costs to be profitable. */ + if (vl == (unsigned) bb_vinfo->bbs[0]->loop_father->num) + continue; + if (dump_enabled_p ()) { dump_printf_loc (MSG_NOTE, vect_location, - "Cost model analysis for part in loop %d:\n", sl); + "Cost model analysis for part in loop %d:\n", + std::min (sl, vl)); dump_printf (MSG_NOTE, " Vector cost: %d\n", vec_inside_cost + vec_outside_cost); dump_printf (MSG_NOTE, " Scalar cost: %d\n", scalar_cost); @@ -9929,15 +9958,21 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo, if (vec_outside_cost + vec_inside_cost > scalar_cost) profitable = false; } - if (profitable && vi < li_vector_costs.length ()) + + if (dump_enabled_p ()) { - if (dump_enabled_p ()) - dump_printf_loc (MSG_NOTE, vect_location, - "Excess vector cost for part in loop %d:\n", - li_vector_costs[vi].first); - profitable = false; + dump_printf_loc (MSG_NOTE, vect_location, + "Cost model analysis for whole subgraph:\n"); + dump_printf (MSG_NOTE, " Vector cost: %d\n", + total_vec_inside_cost + total_vec_outside_cost); + dump_printf (MSG_NOTE, " Scalar cost: %d\n", total_scalar_cost); } + /* For the case where the outermost region had no scalar cost require + overall profitability. */ + if (total_vec_outside_cost + total_vec_inside_cost > total_scalar_cost) + profitable = false; + /* Unset visited flag. This is delayed when the subgraph is profitable and we process the loop for remaining unvectorized if-converted code. */ if (!orig_loop || !profitable)