From patchwork Wed Sep 2 13:49:10 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 142714 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DE29A4BA23C1 for ; Wed, 2 Sep 2026 13:50:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DE29A4BA23C1 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=h3hIJsBq; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=3BBBEcSV; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=fWySwRwX; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=pco/OU7s X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2a07:de40:b251:101:10:150:64:1]) by sourceware.org (Postfix) with ESMTPS id 2B8544BA2E18 for ; Wed, 2 Sep 2026 13:49:20 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2B8544BA2E18 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2B8544BA2E18 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2a07:de40:b251:101:10:150:64:1 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788356960; cv=none; b=BpZHN3EqkYPBLFjrHkp0oB/qIu6yjCy/Vi0W61LzcYbEfqiFOl0yrplVlykLz3pj1yo4+oal+jPKPTkPlBtogdw2orcKzdfvGkGrz03qKTM233adovZKbES2tsHn2+Wc2P2b/gTAgch2JPFCWq/pRtRH03L5vfiwjZMLif2QkA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788356960; c=relaxed/simple; bh=GioDusE0cgdFnWby1iErhNy5yvsjnazhzofj22DBqGM=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:Message-ID:MIME-Version; b=pvH8+tcAVrQFA5dqOEdFUcqEoQ65s+xbS2+fhMjbPaGsF961yhrhw+ffE0OJSH4JrzzTl9wZe1SvMmLKeFnGeorUoIjAH2RBEBFBxst7W5OovJfaBcTlkFkoel97Ch4QygRxF2MgKE2sPsY3claq7FtDppvBQWJ6/AJAy7PsK0E= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=h3hIJsBq; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=3BBBEcSV; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=fWySwRwX; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=pco/OU7s DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2B8544BA2E18 Received: from murzim.nue2.suse.org (unknown [10.168.4.243]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E59EB21D40; Wed, 2 Sep 2026 13:49:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1788356955; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=LjkNO1TRuoBeqXsL5xY6rbq9ZaiRk47zh+6OzYZaZyo=; b=h3hIJsBqyHioZ/3wZAG3+dQ4LYmaKWFEpoy09D0HGqqC0gO8BzmIlzl+x1hxu8kXoCVrOQ wmGeM1qLw+PKsI/sVhsWLXRb21/ntt0L6p7wjO0Pyq6mbd7j0uap7GvovT0WkO164A8Po/ 9AqCkusjaFDyJN0FRa++y5k8BYv1LmE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1788356955; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=LjkNO1TRuoBeqXsL5xY6rbq9ZaiRk47zh+6OzYZaZyo=; b=3BBBEcSV1p3VoWbhEDShhZnH/dQSpJRXRcf0yX5u8xO6TBXtQvEFCe3RnzlL8JhCWbnEYk mVYGFaJGS838gBCg== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1788356950; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=LjkNO1TRuoBeqXsL5xY6rbq9ZaiRk47zh+6OzYZaZyo=; b=fWySwRwXhTmrOfHu1vBJQGFQIHqonl1X3DLGm8ArUtd/V1KOwINNoXJAqjugYC8uTBF7+2 wr1s8d5Dqa1uKnkYax4AzPGFTqUuyEsBJ7NyHvrYmsgWqBcNZLqWlsAuOWNfKKQjIUk6Rg MJtGDcI6ACZYCMB3+9urGTc74eIBg6Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1788356950; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=LjkNO1TRuoBeqXsL5xY6rbq9ZaiRk47zh+6OzYZaZyo=; b=pco/OU7sD7+1srgAXxFZzjm4HJCH4lNlfZm/nCJbECbN+ZDJ7cOJC2mzzUS9PRr1Y16qUR NGsrM3Vs4Pbj0FAw== Date: Wed, 2 Sep 2026 15:49:10 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org cc: rdapp.gcc@gmail.com, tamar.christina@arm.com, hongtao.liu@intel.com Subject: [PATCH] [x86] move CTOR gpr<->xmm move costing to add_slp_cost overload Message-ID: MIME-Version: 1.0 X-Spam-Level: X-Spam-Score: -2.80 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; URIBL_BLOCKED(0.00)[murzim.nue2.suse.org:helo,fhfr.qr:mid,tree-vect-slp.cc:url]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,arm.com,intel.com]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[4] X-Spam-Status: No, score=-11.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org The vectorizer currently works around the fact that the x86 backend in its add_stmt_cost hook walks all SLP node scalar defs to perform gpr<->xmm move costing when costing vector splat/constructor costing for invariants by aoviding to pass the SLP node to each part it is costing. Say for the integer vector { 0, 0, 0, 0, a, b, c, d } and V4SImode vect_prologue_cost_for_slp creates two cost entries, one for the V4SI { 0, 0, 0, 0 } (a load) and one for { a, b, c, d }, a vec_construct. If node was attached to both we'd currently cost the grp<->xmm moves for a, b, c, d twice. The following removes this workaround and instead applies this costing in the add_slp_cost hook which is invoked only once per SLP node. The actual vec_construct cost is still handled by add_stmt_cost. Bootstrapped and tested on x86_64-unknown-linux-gnu. OK for the x86 part? Tamar/Robin, this is a heads-up, not sure if riscv/aarch64 do some similar tinkering and rely on vect_prologue_cost_for_slp setting 'node' only on one cost element. Eventually I'll get screamed at by the CIs, of course. Richard. * config/i386/i386.cc (ix86_vector_costs::add_stmt_cost): Move gpr<->xmm move cost ... (ix86_vector_costs::add_slp_cost): ... here, to new overload of add_slp_cost. * tree-vect-slp.cc (vect_prologue_cost_for_slp): Consistently set SLP node for all cost components to make grouping for the add_slp_cost hook work. (add_slp_costs): Avoid calling vector_costs::add_slp_cost with NULL SLP node. --- gcc/config/i386/i386.cc | 132 ++++++++++++++++++++++------------------ gcc/tree-vect-slp.cc | 21 +++---- 2 files changed, 82 insertions(+), 71 deletions(-) diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc index 9f2196d54de..bd3b12be284 100644 --- a/gcc/config/i386/i386.cc +++ b/gcc/config/i386/i386.cc @@ -26518,6 +26518,7 @@ public: stmt_vec_info stmt_info, slp_tree node, tree vectype, int misalign, vect_cost_model_location where) override; + unsigned int add_slp_cost (slp_tree, const array_slice &); void finish_cost (const vector_costs *) override; bool better_main_loop_than_p (const vector_costs *) const override; bool better_epilogue_loop_than_p (const vector_costs *other, @@ -27111,11 +27112,74 @@ ix86_vector_costs::add_stmt_cost (int count, vect_cost_for_stmt kind, stmt_cost *= (GET_MODE_BITSIZE (TYPE_MODE (ls_type)) / GET_MODE_BITSIZE (TYPE_MODE (ls_eltype)) + 1); } - else if ((kind == vec_construct || kind == scalar_to_vec) - && node - && SLP_TREE_DEF_TYPE (node) == vect_external_def) + if (stmt_cost == -1) + stmt_cost = ix86_default_vector_cost (kind, mode); + + /* BIT_FIELD_REF with count 0 costs 0 in body. */ + if (kind == vec_perm && vectype && count != 0) + { + unsigned vec_size = GET_MODE_SIZE (TYPE_MODE (vectype)); + unsigned nunits = TYPE_VECTOR_SUBPARTS (vectype); + unsigned *num_vec_perm = NULL; + + if (vec_size == 32) + num_vec_perm = m_num_avx256_vec_perm; + else if (vec_size == 64) + num_vec_perm = m_num_avx512_vec_perm; + + if (num_vec_perm && ix86_count_cross_lane_perm_p (m_vinfo, node, nunits)) + { + num_vec_perm[where] += count; + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + "Detected avx%u cross-lane permutation: ", vec_size * 8); + if (stmt_info) + print_gimple_expr (dump_file, stmt_info->stmt, 0, TDF_SLIM); + fprintf (dump_file, " \n"); + } + } + } + + /* Penalize DFmode vector operations for Bonnell. */ + if (TARGET_CPU_P (BONNELL) && kind == vector_stmt + && vectype && GET_MODE_INNER (TYPE_MODE (vectype)) == DFmode) + stmt_cost *= 5; /* FIXME: The value here is arbitrary. */ + + /* Statements in an inner loop relative to the loop being + vectorized are weighted more heavily. The value here is + arbitrary and could potentially be improved with analysis. */ + retval = adjust_cost_for_freq (stmt_info, where, count * stmt_cost); + + /* We need to multiply all vector stmt cost by 1.7 (estimated cost) + for Silvermont as it has out of order integer pipeline and can execute + 2 scalar instruction per tick, but has in order SIMD pipeline. */ + if ((TARGET_CPU_P (SILVERMONT) || TARGET_CPU_P (GOLDMONT) + || TARGET_CPU_P (GOLDMONT_PLUS) || TARGET_CPU_P (INTEL)) + && stmt_info && stmt_info->stmt) + { + tree lhs_op = gimple_get_lhs (stmt_info->stmt); + if (lhs_op && TREE_CODE (TREE_TYPE (lhs_op)) == INTEGER_TYPE) + retval = (retval * 17) / 10; + } + + m_costs[where] += retval; + + return retval; +} + +unsigned +ix86_vector_costs::add_slp_cost (slp_tree node, + const array_slice &parts) +{ + int stmt_cost = 0; + + /* For vector construction account for the cost of moving data between + GRP and XMM. As we are looking at the SLP nodes elements, avoid + duplicate costs by doing this in add_slp_cost, leaving the actual + splat/ctor cost to add_stmt_cost. */ + if (SLP_TREE_DEF_TYPE (node) == vect_external_def) { - stmt_cost = ix86_default_vector_cost (kind, mode); unsigned i; tree op; FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_OPS (node), i, op) @@ -27153,7 +27217,7 @@ ix86_vector_costs::add_stmt_cost (int count, vect_cost_for_stmt kind, ; else { - if (fp) + if (FLOAT_TYPE_P (TREE_TYPE (op))) { /* Scalar FP values residing in x87 registers need to be spilled and reloaded. */ @@ -27166,11 +27230,11 @@ ix86_vector_costs::add_stmt_cost (int count, vect_cost_for_stmt kind, + ix86_cost->sse_load[sse_store_index (mode2)]); stmt_cost += COSTS_N_INSNS (cost) / 2; } - m_num_sse_needed[where]++; + m_num_sse_needed[vect_prologue]++; } else { - m_num_gpr_needed[where]++; + m_num_gpr_needed[vect_prologue]++; stmt_cost += COSTS_N_INSNS (ix86_cost->integer_to_sse) / 2; } @@ -27179,61 +27243,11 @@ ix86_vector_costs::add_stmt_cost (int count, vect_cost_for_stmt kind, FOR_EACH_VEC_ELT (SLP_TREE_SCALAR_OPS (node), i, op) if (TREE_CODE (op) == SSA_NAME) TREE_VISITED (op) = 0; - } - if (stmt_cost == -1) - stmt_cost = ix86_default_vector_cost (kind, mode); - - /* BIT_FIELD_REF with count 0 costs 0 in body. */ - if (kind == vec_perm && vectype && count != 0) - { - unsigned vec_size = GET_MODE_SIZE (TYPE_MODE (vectype)); - unsigned nunits = TYPE_VECTOR_SUBPARTS (vectype); - unsigned *num_vec_perm = NULL; - - if (vec_size == 32) - num_vec_perm = m_num_avx256_vec_perm; - else if (vec_size == 64) - num_vec_perm = m_num_avx512_vec_perm; - if (num_vec_perm && ix86_count_cross_lane_perm_p (m_vinfo, node, nunits)) - { - num_vec_perm[where] += count; - if (dump_file && (dump_flags & TDF_DETAILS)) - { - fprintf (dump_file, - "Detected avx%u cross-lane permutation: ", vec_size * 8); - if (stmt_info) - print_gimple_expr (dump_file, stmt_info->stmt, 0, TDF_SLIM); - fprintf (dump_file, " \n"); - } - } + m_costs[vect_prologue] += stmt_cost; } - /* Penalize DFmode vector operations for Bonnell. */ - if (TARGET_CPU_P (BONNELL) && kind == vector_stmt - && vectype && GET_MODE_INNER (TYPE_MODE (vectype)) == DFmode) - stmt_cost *= 5; /* FIXME: The value here is arbitrary. */ - - /* Statements in an inner loop relative to the loop being - vectorized are weighted more heavily. The value here is - arbitrary and could potentially be improved with analysis. */ - retval = adjust_cost_for_freq (stmt_info, where, count * stmt_cost); - - /* We need to multiply all vector stmt cost by 1.7 (estimated cost) - for Silvermont as it has out of order integer pipeline and can execute - 2 scalar instruction per tick, but has in order SIMD pipeline. */ - if ((TARGET_CPU_P (SILVERMONT) || TARGET_CPU_P (GOLDMONT) - || TARGET_CPU_P (GOLDMONT_PLUS) || TARGET_CPU_P (INTEL)) - && stmt_info && stmt_info->stmt) - { - tree lhs_op = gimple_get_lhs (stmt_info->stmt); - if (lhs_op && TREE_CODE (TREE_TYPE (lhs_op)) == INTEGER_TYPE) - retval = (retval * 17) / 10; - } - - m_costs[where] += retval; - - return retval; + return stmt_cost + vector_costs::add_slp_cost (node, parts); } void diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index b8096f9256d..3e1bfd000f1 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -9005,7 +9005,6 @@ vect_prologue_cost_for_slp (slp_tree node, unsigned nvectors, } /* ??? We're just tracking whether vectors in a single node are the same. Ideally we'd do something more global. */ - bool passed = false; for (unsigned int start : starts) { vect_cost_for_stmt kind; @@ -9015,15 +9014,8 @@ vect_prologue_cost_for_slp (slp_tree node, unsigned nvectors, kind = scalar_to_vec; else kind = vec_construct; - /* The target cost hook has no idea which part of the SLP node - we are costing so avoid passing it down more than once. Pass - it to the first vec_construct or scalar_to_vec part since for those - the x86 backend tries to account for GPR to XMM register moves. */ - record_stmt_cost (cost_vec, 1, kind, nullptr, - (kind != vector_load && !passed) ? node : nullptr, + record_stmt_cost (cost_vec, 1, kind, nullptr, node, vectype, 0, vect_prologue); - if (kind != vector_load) - passed = true; } } @@ -9397,9 +9389,14 @@ add_slp_costs (vector_costs *costs, stmt_vector_for_cost& cost_vec) while (end < cost_vec.length () && cost_vec[start].node == cost_vec[end].node) end++; - costs->add_slp_cost (cost_vec[start].node, - array_slice - (cost_vec.begin () + start, end - start)); + if (cost_vec[start].node) + costs->add_slp_cost (cost_vec[start].node, + array_slice + (cost_vec.begin () + start, end - start)); + else + costs->vector_costs::add_slp_cost (cost_vec[start].node, + array_slice + (cost_vec.begin () + start, end - start)); start = end; } }