From patchwork Wed Oct 30 18:48:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 99843 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 22A9E3858290 for ; Wed, 30 Oct 2024 18:51:02 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id 500033858429 for ; Wed, 30 Oct 2024 18:49:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 500033858429 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 500033858429 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1730314168; cv=none; b=ptuaqtBV/hTPRblyp7LQ+kV0Nz/UEZNiT/DtX8VM6EDJ4EMx5UONGs8mkVqr7DN03YlZ2GUPvo5JtX1cMyqOhc7x7nhtZtoChFxUHMZyaIfSJ6JQ91hHxLunMZPeRYJ3iYQ5RXt2VMz7BMt5mqEjDOr8znkYiTMrSGw9LHdeRs0= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1730314168; c=relaxed/simple; bh=CtCHLdT9orsvQif0ypEuo+jqE84cQ4Gfj+6/0OJL9dA=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=HZP8cpeMjrH5KdHbanIdhQfWWtRayFOTxPTaEyK+/Vu/aMRWcRgt+9cMGBoZde0J9ib/z7NRY7W615Q9Pi93GcKddJlY2PUa1oWsQ9TbV2aR+FeJyBb4BJjnAELKg6gfiZUQHiNqoWluczqs9FxtILD+7ZK9hIjAeTVIQ5DdjpI= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CECFF1A00; Wed, 30 Oct 2024 11:49:51 -0700 (PDT) Received: from e121540-lin.manchester.arm.com (e121540-lin.manchester.arm.com [10.32.110.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2D7DD3F528; Wed, 30 Oct 2024 11:49:21 -0700 (PDT) From: Richard Sandiford To: richard.earnshaw@arm.com, ktkachov@nvidia.com, gcc-patches@gcc.gnu.org Cc: Richard Sandiford Subject: [PATCH 2/3] aarch64: Record separate streaming and non-streaming ISA requirements Date: Wed, 30 Oct 2024 18:48:29 +0000 Message-Id: <20241030184830.3634301-3-richard.sandiford@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241030184830.3634301-1-richard.sandiford@arm.com> References: <20241030184830.3634301-1-richard.sandiford@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-18.4 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_NONE, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org For some upcoming extensions, we need to add intrinsics whose ISA requirements differ between streaming mode and non-streaming mode. This patch tries to generalise the infrastructure to support that: - Rather than have a single set of feature flags, the patch uses a separate set for sm_off (non-streaming, PSTATE.SM==0) and sm_on (streaming, PSTATE.SM==1). - The sm_off set is zero if the intrinsic is streaming-only. Otherwise it is AARCH64_FL_SM_OFF | . - Similarly, the sm_on set is zero if the intrinsic is non-streaming-only. Otherwise it is AARCH64_FL_SM_ON | . AARCH64_FL_SME is taken as given in streaming mode. - Streaming-compatible code must satisfy both sets of requirements. There should be no functional change. gcc/ * config.gcc (aarch64*-*-*): Add aarch64-protos.h to target_gtfiles. * config/aarch64/aarch64-protos.h (aarch64_required_extensions): New structure. (aarch64_check_required_extensions): Change the type of the required_extensions parameter from aarch64_feature_flags to aarch64_required_extensions. * config/aarch64/aarch64-sve-builtins.h (function_builder::add_unique_function): Likewise. (function_builder::add_overloaded_function): Likewise. (function_builder::get_attributes): Likewise. (function_builder::add_function): Likewise. (function_group_info): Change the type of required_extensions in the same way. * config/aarch64/aarch64-builtins.cc (aarch64_pragma_builtins_data::required_extensions): Change the type from aarch64_feature_flags to aarch64_required_extensions. (aarch64_check_required_extensions): Likewise change the type of the required_extensions parameter. Separate the requirements for non-streaming mode and streaming mode, ORing them together for streaming-compatible mode. (aarch64_general_required_extensions): New function. (aarch64_general_check_builtin_call): Use it. * config/aarch64/aarch64-sve-builtins.cc (registered_function::required_extensions): Change the type from aarch64_feature_flags to aarch64_required_extensions. (DEF_NEON_SVE_FUNCTION, DEF_SME_ZA_FUNCTION_GS): Update accordingly. (function_builder::get_attributes): Change the type of the required_extensions parameter from aarch64_feature_flags to aarch64_required_extensions. (function_builder::add_function): Likewise. (function_builder::add_unique_function): Likewise. (function_builder::add_overloaded_function): Likewise. * config/aarch64/aarch64-simd-pragma-builtins.def: Update REQUIRED_EXTENSIONS definitions to use aarch64_required_extensions. * config/aarch64/aarch64-sve-builtins-base.def: Likewise. * config/aarch64/aarch64-sve-builtins-sme.def: Likewise. * config/aarch64/aarch64-sve-builtins-sve2.def: Likewise. --- gcc/config.gcc | 2 +- gcc/config/aarch64/aarch64-builtins.cc | 122 ++++++++++-------- gcc/config/aarch64/aarch64-protos.h | 87 ++++++++++++- .../aarch64/aarch64-simd-pragma-builtins.def | 2 +- .../aarch64/aarch64-sve-builtins-base.def | 26 ++-- .../aarch64/aarch64-sve-builtins-sme.def | 30 ++--- .../aarch64/aarch64-sve-builtins-sve2.def | 41 ++---- gcc/config/aarch64/aarch64-sve-builtins.cc | 51 +++++--- gcc/config/aarch64/aarch64-sve-builtins.h | 13 +- 9 files changed, 226 insertions(+), 148 deletions(-) diff --git a/gcc/config.gcc b/gcc/config.gcc index e2ed3b309cc..c3531e56c9d 100644 --- a/gcc/config.gcc +++ b/gcc/config.gcc @@ -352,7 +352,7 @@ aarch64*-*-*) cxx_target_objs="aarch64-c.o" d_target_objs="aarch64-d.o" extra_objs="aarch64-builtins.o aarch-common.o aarch64-sve-builtins.o aarch64-sve-builtins-shapes.o aarch64-sve-builtins-base.o aarch64-sve-builtins-sve2.o aarch64-sve-builtins-sme.o cortex-a57-fma-steering.o aarch64-speculation.o falkor-tag-collision-avoidance.o aarch-bti-insert.o aarch64-cc-fusion.o aarch64-early-ra.o aarch64-ldp-fusion.o" - target_gtfiles="\$(srcdir)/config/aarch64/aarch64-builtins.h \$(srcdir)/config/aarch64/aarch64-builtins.cc \$(srcdir)/config/aarch64/aarch64-sve-builtins.h \$(srcdir)/config/aarch64/aarch64-sve-builtins.cc" + target_gtfiles="\$(srcdir)/config/aarch64/aarch64-protos.h \$(srcdir)/config/aarch64/aarch64-builtins.h \$(srcdir)/config/aarch64/aarch64-builtins.cc \$(srcdir)/config/aarch64/aarch64-sve-builtins.h \$(srcdir)/config/aarch64/aarch64-sve-builtins.cc" target_has_targetm_common=yes ;; alpha*-*-*) diff --git a/gcc/config/aarch64/aarch64-builtins.cc b/gcc/config/aarch64/aarch64-builtins.cc index 480ac223d86..97bde7c15d3 100644 --- a/gcc/config/aarch64/aarch64-builtins.cc +++ b/gcc/config/aarch64/aarch64-builtins.cc @@ -1595,7 +1595,8 @@ enum class aarch64_builtin_signatures #undef ENTRY #define ENTRY(N, S, M, U) \ - {#N, aarch64_builtin_signatures::S, E_##M##mode, U, REQUIRED_EXTENSIONS}, + {#N, aarch64_builtin_signatures::S, E_##M##mode, U, \ + aarch64_required_extensions::REQUIRED_EXTENSIONS}, /* Initialize pragma builtins. */ @@ -1605,7 +1606,7 @@ struct aarch64_pragma_builtins_data aarch64_builtin_signatures signature; machine_mode mode; int unspec; - aarch64_feature_flags required_extensions; + aarch64_required_extensions required_extensions; }; static aarch64_pragma_builtins_data aarch64_pragma_builtins[] = { @@ -2333,18 +2334,40 @@ aarch64_report_missing_registers (location_t location, tree fndecl) reported_missing_registers_p = true; } -/* Check whether all the AARCH64_FL_* values in REQUIRED_EXTENSIONS are - enabled, given that those extensions are required for function FNDECL. - Report an error against LOCATION if not. */ +/* Check whether the requirements in REQUIRED_EXTENSIONS are met, given that + those requirements come from calling function FNDECL. Report an error + against LOCATION if not. */ bool aarch64_check_required_extensions (location_t location, tree fndecl, - aarch64_feature_flags required_extensions) + aarch64_required_extensions + required_extensions) { - if ((required_extensions & ~aarch64_isa_flags) == 0) - return true; + aarch64_feature_flags sm_state_extensions = 0; + if (!TARGET_STREAMING) + { + if (required_extensions.sm_off == 0) + { + error_at (location, "ACLE function %qD can only be called when" + " SME streaming mode is enabled", fndecl); + return false; + } + sm_state_extensions |= required_extensions.sm_off & ~AARCH64_FL_SM_OFF; + } + if (!TARGET_NON_STREAMING) + { + if (required_extensions.sm_on == 0) + { + error_at (location, "ACLE function %qD cannot be called when" + " SME streaming mode is enabled", fndecl); + return false; + } + sm_state_extensions |= required_extensions.sm_on & ~AARCH64_FL_SM_ON; + } - auto missing_extensions = required_extensions & ~aarch64_asm_isa_flags; + if ((sm_state_extensions & ~aarch64_isa_flags) == 0) + return true; + auto missing_extensions = sm_state_extensions & ~aarch64_asm_isa_flags; if (missing_extensions == 0) { /* All required extensions are enabled in aarch64_asm_isa_flags, so the @@ -2353,20 +2376,6 @@ aarch64_check_required_extensions (location_t location, tree fndecl, return false; } - if (missing_extensions & AARCH64_FL_SM_OFF) - { - error_at (location, "ACLE function %qD cannot be called when" - " SME streaming mode is enabled", fndecl); - return false; - } - - if (missing_extensions & AARCH64_FL_SM_ON) - { - error_at (location, "ACLE function %qD can only be called when" - " SME streaming mode is enabled", fndecl); - return false; - } - if (missing_extensions & AARCH64_FL_ZA_ON) { error_at (location, "ACLE function %qD can only be called from" @@ -2392,12 +2401,47 @@ aarch64_check_required_extensions (location_t location, tree fndecl, gcc_unreachable (); } +/* Return the ISA extensions required by function CODE. */ +static aarch64_required_extensions +aarch64_general_required_extensions (unsigned int code) +{ + using ext = aarch64_required_extensions; + switch (code) + { + case AARCH64_TME_BUILTIN_TSTART: + case AARCH64_TME_BUILTIN_TCOMMIT: + case AARCH64_TME_BUILTIN_TTEST: + case AARCH64_TME_BUILTIN_TCANCEL: + return ext::streaming_compatible (AARCH64_FL_TME); + + case AARCH64_LS64_BUILTIN_LD64B: + case AARCH64_LS64_BUILTIN_ST64B: + case AARCH64_LS64_BUILTIN_ST64BV: + case AARCH64_LS64_BUILTIN_ST64BV0: + return ext::streaming_compatible (AARCH64_FL_LS64); + + default: + if (code >= AARCH64_MEMTAG_BUILTIN_START + && code <= AARCH64_MEMTAG_BUILTIN_END) + return ext::streaming_compatible (AARCH64_FL_MEMTAG); + + if (auto builtin_data = aarch64_get_pragma_builtin (code)) + return builtin_data->required_extensions; + } + return ext::streaming_compatible (0); +} + bool aarch64_general_check_builtin_call (location_t location, vec, - unsigned int code, tree fndecl, - unsigned int nargs ATTRIBUTE_UNUSED, tree *args) + unsigned int code, tree fndecl, + unsigned int nargs ATTRIBUTE_UNUSED, + tree *args) { tree decl = aarch64_builtin_decls[code]; + auto required_extensions = aarch64_general_required_extensions (code); + if (!aarch64_check_required_extensions (location, decl, required_extensions)) + return false; + switch (code) { case AARCH64_RSR: @@ -2423,34 +2467,6 @@ aarch64_general_check_builtin_call (location_t location, vec, } break; } - - case AARCH64_TME_BUILTIN_TSTART: - case AARCH64_TME_BUILTIN_TCOMMIT: - case AARCH64_TME_BUILTIN_TTEST: - case AARCH64_TME_BUILTIN_TCANCEL: - return aarch64_check_required_extensions (location, decl, - AARCH64_FL_TME); - - case AARCH64_LS64_BUILTIN_LD64B: - case AARCH64_LS64_BUILTIN_ST64B: - case AARCH64_LS64_BUILTIN_ST64BV: - case AARCH64_LS64_BUILTIN_ST64BV0: - return aarch64_check_required_extensions (location, decl, - AARCH64_FL_LS64); - - default: - break; - } - - if (code >= AARCH64_MEMTAG_BUILTIN_START - && code <= AARCH64_MEMTAG_BUILTIN_END) - return aarch64_check_required_extensions (location, decl, - AARCH64_FL_MEMTAG); - - if (auto builtin_data = aarch64_get_pragma_builtin (code)) - { - auto flags = builtin_data->required_extensions; - return aarch64_check_required_extensions (location, decl, flags); } return true; diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index 9be64913091..05caad5e2fe 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -744,6 +744,91 @@ private: bool m_old_general_regs_only; }; +/* Represents the ISA requirements of an intrinsic function, or of some + other similar operation. It stores separate feature flags for + non-streaming mode and for streaming-mode; both requirements must + be met in streaming-compatible mode. */ +struct aarch64_required_extensions +{ + /* Return a requirement that includes FLAGS on top of any existing + requirements. */ + inline CONSTEXPR aarch64_required_extensions + and_also (aarch64_feature_flags flags) + { + return { sm_off ? sm_off | flags : 0, + sm_on ? sm_on | flags : 0 }; + } + + /* Require non-streaming mode and the features in FLAGS. */ + static inline CONSTEXPR aarch64_required_extensions + nonstreaming_only (aarch64_feature_flags flags) + { + return { AARCH64_FL_SM_OFF | flags, 0 }; + } + + /* Likewise, and also require SVE. */ + static inline CONSTEXPR aarch64_required_extensions + nonstreaming_sve (aarch64_feature_flags flags) + { + return nonstreaming_only (AARCH64_FL_SVE | flags); + } + + /* Allow both streaming and non-streaming mode, requiring the features + in FLAGS for both cases. */ + static inline CONSTEXPR aarch64_required_extensions + streaming_compatible (aarch64_feature_flags flags) + { + return { AARCH64_FL_SM_OFF | flags, AARCH64_FL_SM_ON | flags }; + } + + /* Likewise, and also require SVE for non-streaming mode. */ + static inline CONSTEXPR aarch64_required_extensions + ssve (aarch64_feature_flags flags) + { + return streaming_compatible (AARCH64_FL_SVE | flags, flags); + } + + /* Allow both streaming and non-streaming mode, requiring the features + in SM_OFF for non-streaming mode and the features in SM_ON for + streaming mode. */ + static inline CONSTEXPR aarch64_required_extensions + streaming_compatible (aarch64_feature_flags sm_off, + aarch64_feature_flags sm_on) + { + return { AARCH64_FL_SM_OFF | sm_off, AARCH64_FL_SM_ON | sm_on }; + } + + /* Likewise, and also require SVE for non-streaming mode. */ + static inline CONSTEXPR aarch64_required_extensions + sve_and_sme (aarch64_feature_flags sm_off, aarch64_feature_flags sm_on) + { + return streaming_compatible (AARCH64_FL_SVE | sm_off, sm_on); + } + + /* Require streaming mode and the features in FLAGS. */ + static inline CONSTEXPR aarch64_required_extensions + streaming_only (aarch64_feature_flags flags) + { + return { 0, AARCH64_FL_SM_ON | flags }; + } + + /* The ISA requirements in non-streaming mode, or 0 if the operation + is only allowed in streaming mode. When this field is nonzero, + it always includes AARCH64_FL_SM_OFF. */ + aarch64_feature_flags sm_off; + + /* The ISA requirements in streaming mode, or 0 if the operation is only + allowed in non-streaming mode. When this field is nonzero, + it always includes AARCH64_FL_SM_ON. + + This field should not normally include AARCH64_FL_SME, since we + would not be in streaming mode if SME wasn't supported. Excluding + AARCH64_FL_SME makes it easier to handle streaming-compatible rules + since (for example) svadd_x should be available in streaming-compatible + functions even without +sme. */ + aarch64_feature_flags sm_on; +}; + void aarch64_post_cfi_startproc (void); poly_int64 aarch64_initial_elimination_offset (unsigned, unsigned); int aarch64_get_condition_code (rtx); @@ -1015,7 +1100,7 @@ void handle_arm_acle_h (void); void handle_arm_neon_h (void); bool aarch64_check_required_extensions (location_t, tree, - aarch64_feature_flags); + aarch64_required_extensions); bool aarch64_general_check_builtin_call (location_t, vec, unsigned int, tree, unsigned int, tree *); diff --git a/gcc/config/aarch64/aarch64-simd-pragma-builtins.def b/gcc/config/aarch64/aarch64-simd-pragma-builtins.def index 9d530fc45d4..d66642eaa0a 100644 --- a/gcc/config/aarch64/aarch64-simd-pragma-builtins.def +++ b/gcc/config/aarch64/aarch64-simd-pragma-builtins.def @@ -27,7 +27,7 @@ ENTRY (NAME##q_f64, SIGNATURE, V2DF, UNSPEC) // faminmax -#define REQUIRED_EXTENSIONS AARCH64_FL_FAMINMAX +#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_FAMINMAX) ENTRY_VHSDF (vamax, binary, UNSPEC_FAMAX) ENTRY_VHSDF (vamin, binary, UNSPEC_FAMIN) #undef REQUIRED_EXTENSIONS diff --git a/gcc/config/aarch64/aarch64-sve-builtins-base.def b/gcc/config/aarch64/aarch64-sve-builtins-base.def index d45f8f28ab8..edfe2574507 100644 --- a/gcc/config/aarch64/aarch64-sve-builtins-base.def +++ b/gcc/config/aarch64/aarch64-sve-builtins-base.def @@ -17,7 +17,7 @@ along with GCC; see the file COPYING3. If not see . */ -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE +#define REQUIRED_EXTENSIONS ssve (0) DEF_SVE_FUNCTION (svabd, binary_opt_n, all_arith, mxz) DEF_SVE_FUNCTION (svabs, unary, all_float_and_signed, mxz) DEF_SVE_FUNCTION (svacge, compare_opt_n, all_float, implicit) @@ -261,7 +261,7 @@ DEF_SVE_FUNCTION (svzip2, binary, all_data, none) DEF_SVE_FUNCTION (svzip2, binary_pred, all_pred, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE | AARCH64_FL_SM_OFF +#define REQUIRED_EXTENSIONS nonstreaming_sve (0) DEF_SVE_FUNCTION (svadda, fold_left, all_float, implicit) DEF_SVE_FUNCTION (svadrb, adr_offset, none, none) DEF_SVE_FUNCTION (svadrd, adr_index, none, none) @@ -327,7 +327,7 @@ DEF_SVE_FUNCTION (svtssel, binary_uint, all_float, none) DEF_SVE_FUNCTION (svwrffr, setffr, none, implicit) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE | AARCH64_FL_BF16 +#define REQUIRED_EXTENSIONS ssve (AARCH64_FL_BF16) DEF_SVE_FUNCTION (svbfdot, ternary_bfloat_opt_n, s_float, none) DEF_SVE_FUNCTION (svbfdot_lane, ternary_bfloat_lanex2, s_float, none) DEF_SVE_FUNCTION (svbfmlalb, ternary_bfloat_opt_n, s_float, none) @@ -338,35 +338,27 @@ DEF_SVE_FUNCTION (svcvt, unary_convertxn, cvt_bfloat, mxz) DEF_SVE_FUNCTION (svcvtnt, unary_convert_narrowt, cvt_bfloat, mx) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_BF16 \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_BF16) DEF_SVE_FUNCTION (svbfmmla, ternary_bfloat, s_float, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE | AARCH64_FL_I8MM +#define REQUIRED_EXTENSIONS ssve (AARCH64_FL_I8MM) DEF_SVE_FUNCTION (svsudot, ternary_intq_uintq_opt_n, s_signed, none) DEF_SVE_FUNCTION (svsudot_lane, ternary_intq_uintq_lane, s_signed, none) DEF_SVE_FUNCTION (svusdot, ternary_uintq_intq_opt_n, s_signed, none) DEF_SVE_FUNCTION (svusdot_lane, ternary_uintq_intq_lane, s_signed, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_I8MM \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_I8MM) DEF_SVE_FUNCTION (svmmla, mmla, s_integer, none) DEF_SVE_FUNCTION (svusmmla, ternary_uintq_intq, s_signed, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_F32MM \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_F32MM) DEF_SVE_FUNCTION (svmmla, mmla, s_float, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_F64MM \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_F64MM) DEF_SVE_FUNCTION (svld1ro, load_replicate, all_data, implicit) DEF_SVE_FUNCTION (svmmla, mmla, d_float, none) DEF_SVE_FUNCTION (svtrn1q, binary, all_data, none) @@ -377,7 +369,7 @@ DEF_SVE_FUNCTION (svzip1q, binary, all_data, none) DEF_SVE_FUNCTION (svzip2q, binary, all_data, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE | AARCH64_FL_FAMINMAX +#define REQUIRED_EXTENSIONS ssve (AARCH64_FL_FAMINMAX) DEF_SVE_FUNCTION (svamax, binary_opt_single_n, all_float, mxz) DEF_SVE_FUNCTION (svamin, binary_opt_single_n, all_float, mxz) #undef REQUIRED_EXTENSIONS diff --git a/gcc/config/aarch64/aarch64-sve-builtins-sme.def b/gcc/config/aarch64/aarch64-sve-builtins-sme.def index 416df0b3637..bc2c3323636 100644 --- a/gcc/config/aarch64/aarch64-sve-builtins-sme.def +++ b/gcc/config/aarch64/aarch64-sve-builtins-sme.def @@ -32,12 +32,12 @@ DEF_SME_ZA_FUNCTION_GS (NAME, SHAPE, TYPES, none, PREDS) #endif -#define REQUIRED_EXTENSIONS 0 +#define REQUIRED_EXTENSIONS streaming_compatible (0) DEF_SME_FUNCTION (arm_has_sme, bool_inherent, none, none) DEF_SME_FUNCTION (arm_in_streaming_mode, bool_inherent, none, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SME +#define REQUIRED_EXTENSIONS streaming_compatible (AARCH64_FL_SME) DEF_SME_FUNCTION (svcntsb, count_inherent, none, none) DEF_SME_FUNCTION (svcntsd, count_inherent, none, none) DEF_SME_FUNCTION (svcntsh, count_inherent, none, none) @@ -49,7 +49,7 @@ DEF_SME_ZA_FUNCTION (svzero, inherent_za, za, none) DEF_SME_ZA_FUNCTION (svzero_mask, inherent_mask_za, za, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SME | AARCH64_FL_SM_ON +#define REQUIRED_EXTENSIONS streaming_only (0) DEF_SME_ZA_FUNCTION (svaddha, unary_za_m, za_s_integer, za_m) DEF_SME_ZA_FUNCTION (svaddva, unary_za_m, za_s_integer, za_m) DEF_SME_ZA_FUNCTION (svld1_hor, load_za, all_za, none) @@ -70,9 +70,7 @@ DEF_SME_ZA_FUNCTION (svwrite_hor, write_za_m, za_all_data, za_m) DEF_SME_ZA_FUNCTION (svwrite_ver, write_za_m, za_all_data, za_m) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SME \ - | AARCH64_FL_SME_I16I64 \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME_I16I64) DEF_SME_ZA_FUNCTION (svaddha, unary_za_m, za_d_integer, za_m) DEF_SME_ZA_FUNCTION (svaddva, unary_za_m, za_d_integer, za_m) DEF_SME_ZA_FUNCTION (svmopa, binary_za_m, mop_i16i64, za_m) @@ -83,14 +81,12 @@ DEF_SME_ZA_FUNCTION (svusmopa, binary_za_int_m, mop_i16i64_unsigned, za_m) DEF_SME_ZA_FUNCTION (svusmops, binary_za_int_m, mop_i16i64_unsigned, za_m) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SME \ - | AARCH64_FL_SME_F64F64 \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME_F64F64) DEF_SME_ZA_FUNCTION (svmopa, binary_za_m, za_d_float, za_m) DEF_SME_ZA_FUNCTION (svmops, binary_za_m, za_d_float, za_m) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS AARCH64_FL_SME | AARCH64_FL_SME2 +#define REQUIRED_EXTENSIONS streaming_compatible (AARCH64_FL_SME2) DEF_SME_FUNCTION (svldr_zt, ldr_zt, none, none) DEF_SME_FUNCTION (svstr_zt, str_zt, none, none) DEF_SME_FUNCTION (svzero_zt, inherent_zt, none, none) @@ -100,7 +96,7 @@ DEF_SME_FUNCTION (svzero_zt, inherent_zt, none, none) which will then be resolved to either an integer function or a floating-point function. They are needed because the integer and floating-point functions have different architecture requirements. */ -#define REQUIRED_EXTENSIONS AARCH64_FL_SME | AARCH64_FL_SME2 | AARCH64_FL_SM_ON +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME2) DEF_SME_ZA_FUNCTION_GS (svadd, unary_za_slice, za_s_data, vg1x24, none) DEF_SME_ZA_FUNCTION_GS (svadd, unary_za_slice, d_za, vg1x24, none) DEF_SME_ZA_FUNCTION_GS (svadd_write, binary_za_slice_opt_single, za_s_integer, @@ -172,10 +168,8 @@ DEF_SME_ZA_FUNCTION_GS (svwrite_hor, write_za, za_bhsd_data, vg24, none) DEF_SME_ZA_FUNCTION_GS (svwrite_ver, write_za, za_bhsd_data, vg24, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SME \ - | AARCH64_FL_SME2 \ - | AARCH64_FL_SME_I16I64 \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME2 \ + | AARCH64_FL_SME_I16I64) DEF_SME_ZA_FUNCTION_GS (svadd, unary_za_slice, za_d_integer, vg1x24, none) DEF_SME_ZA_FUNCTION_GS (svadd_write, binary_za_slice_opt_single, za_d_integer, vg1x24, none) @@ -198,10 +192,8 @@ DEF_SME_ZA_FUNCTION_GS (svvdot_lane, dot_za_slice_lane, za_d_h_integer, vg1x4, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SME \ - | AARCH64_FL_SME2 \ - | AARCH64_FL_SME_F64F64 \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME2 \ + | AARCH64_FL_SME_F64F64) DEF_SME_ZA_FUNCTION_GS (svadd, unary_za_slice, za_d_float, vg1x24, none) DEF_SME_ZA_FUNCTION_GS (svmla, binary_za_slice_opt_single, za_d_float, vg1x24, none) diff --git a/gcc/config/aarch64/aarch64-sve-builtins-sve2.def b/gcc/config/aarch64/aarch64-sve-builtins-sve2.def index 318dfff06f0..345a7621b6f 100644 --- a/gcc/config/aarch64/aarch64-sve-builtins-sve2.def +++ b/gcc/config/aarch64/aarch64-sve-builtins-sve2.def @@ -17,7 +17,7 @@ along with GCC; see the file COPYING3. If not see . */ -#define REQUIRED_EXTENSIONS AARCH64_FL_SVE | AARCH64_FL_SVE2 +#define REQUIRED_EXTENSIONS sve_and_sme (AARCH64_FL_SVE2, 0) DEF_SVE_FUNCTION (svaba, ternary_opt_n, all_integer, none) DEF_SVE_FUNCTION (svabalb, ternary_long_opt_n, hsd_integer, none) DEF_SVE_FUNCTION (svabalt, ternary_long_opt_n, hsd_integer, none) @@ -166,9 +166,7 @@ DEF_SVE_FUNCTION (svwhilewr, compare_ptr, all_data, none) DEF_SVE_FUNCTION (svxar, ternary_shift_right_imm, all_integer, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_SVE2) DEF_SVE_FUNCTION (svhistcnt, binary_to_uint, sd_integer, z) DEF_SVE_FUNCTION (svhistseg, binary_to_uint, b_integer, none) DEF_SVE_FUNCTION (svldnt1_gather, load_gather_sv_restricted, sd_data, implicit) @@ -194,10 +192,8 @@ DEF_SVE_FUNCTION (svstnt1w_scatter, store_scatter_index_restricted, d_integer, i DEF_SVE_FUNCTION (svstnt1w_scatter, store_scatter_offset_restricted, d_integer, implicit) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SVE2_AES \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_SVE2 \ + | AARCH64_FL_SVE2_AES) DEF_SVE_FUNCTION (svaesd, binary, b_unsigned, none) DEF_SVE_FUNCTION (svaese, binary, b_unsigned, none) DEF_SVE_FUNCTION (svaesmc, unary, b_unsigned, none) @@ -206,44 +202,31 @@ DEF_SVE_FUNCTION (svpmullb_pair, binary_opt_n, d_unsigned, none) DEF_SVE_FUNCTION (svpmullt_pair, binary_opt_n, d_unsigned, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SVE2_BITPERM \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_SVE2 \ + | AARCH64_FL_SVE2_BITPERM) DEF_SVE_FUNCTION (svbdep, binary_opt_n, all_unsigned, none) DEF_SVE_FUNCTION (svbext, binary_opt_n, all_unsigned, none) DEF_SVE_FUNCTION (svbgrp, binary_opt_n, all_unsigned, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SVE2_SHA3 \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_SVE2 \ + | AARCH64_FL_SVE2_SHA3) DEF_SVE_FUNCTION (svrax1, binary, d_integer, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SVE2_SM4 \ - | AARCH64_FL_SM_OFF) +#define REQUIRED_EXTENSIONS nonstreaming_sve (AARCH64_FL_SVE2 \ + | AARCH64_FL_SVE2_SM4) DEF_SVE_FUNCTION (svsm4e, binary, s_unsigned, none) DEF_SVE_FUNCTION (svsm4ekey, binary, s_unsigned, none) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SME \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (0) DEF_SVE_FUNCTION (svclamp, clamp, all_integer, none) DEF_SVE_FUNCTION (svpsel_lane, select_pred, all_pred_count, none) DEF_SVE_FUNCTION (svrevd, unary, all_data, mxz) #undef REQUIRED_EXTENSIONS -#define REQUIRED_EXTENSIONS (AARCH64_FL_SVE \ - | AARCH64_FL_SVE2 \ - | AARCH64_FL_SME \ - | AARCH64_FL_SME2 \ - | AARCH64_FL_SM_ON) +#define REQUIRED_EXTENSIONS streaming_only (AARCH64_FL_SME2) DEF_SVE_FUNCTION_GS (svadd, binary_single, all_integer, x24, none) DEF_SVE_FUNCTION (svbfmlslb, ternary_bfloat_opt_n, s_float, none) DEF_SVE_FUNCTION (svbfmlslb_lane, ternary_bfloat_lane, s_float, none) diff --git a/gcc/config/aarch64/aarch64-sve-builtins.cc b/gcc/config/aarch64/aarch64-sve-builtins.cc index ef14f8cd39d..44b7f6edae5 100644 --- a/gcc/config/aarch64/aarch64-sve-builtins.cc +++ b/gcc/config/aarch64/aarch64-sve-builtins.cc @@ -83,9 +83,8 @@ public: /* The decl itself. */ tree decl; - /* The architecture extensions that the function requires, as a set of - AARCH64_FL_* flags. */ - aarch64_feature_flags required_extensions; + /* The architecture extensions that the function requires. */ + aarch64_required_extensions required_extensions; /* True if the decl represents an overloaded function that needs to be resolved by function_resolver. */ @@ -883,11 +882,15 @@ static const predication_index preds_z[] = { PRED_z, NUM_PREDS }; /* Used by SME instructions that always merge into ZA. */ static const predication_index preds_za_m[] = { PRED_za_m, NUM_PREDS }; +#define NONSTREAMING_SVE(X) nonstreaming_only (AARCH64_FL_SVE | (X)) +#define SVE_AND_SME(X, Y) streaming_compatible (AARCH64_FL_SVE | (X), (Y)) +#define SSVE(X) SVE_AND_SME (X, X) + /* A list of all arm_sve.h functions. */ static CONSTEXPR const function_group_info function_groups[] = { #define DEF_SVE_FUNCTION_GS(NAME, SHAPE, TYPES, GROUPS, PREDS) \ { #NAME, &functions::NAME, &shapes::SHAPE, types_##TYPES, groups_##GROUPS, \ - preds_##PREDS, REQUIRED_EXTENSIONS }, + preds_##PREDS, aarch64_required_extensions::REQUIRED_EXTENSIONS }, #include "aarch64-sve-builtins.def" }; @@ -895,7 +898,7 @@ static CONSTEXPR const function_group_info function_groups[] = { static CONSTEXPR const function_group_info neon_sve_function_groups[] = { #define DEF_NEON_SVE_FUNCTION(NAME, SHAPE, TYPES, GROUPS, PREDS) \ { #NAME, &neon_sve_bridge_functions::NAME, &shapes::SHAPE, types_##TYPES, \ - groups_##GROUPS, preds_##PREDS, 0 }, + groups_##GROUPS, preds_##PREDS, aarch64_required_extensions::ssve (0) }, #include "aarch64-neon-sve-bridge-builtins.def" }; @@ -903,10 +906,12 @@ static CONSTEXPR const function_group_info neon_sve_function_groups[] = { static CONSTEXPR const function_group_info sme_function_groups[] = { #define DEF_SME_FUNCTION_GS(NAME, SHAPE, TYPES, GROUPS, PREDS) \ { #NAME, &functions::NAME, &shapes::SHAPE, types_##TYPES, groups_##GROUPS, \ - preds_##PREDS, REQUIRED_EXTENSIONS }, + preds_##PREDS, aarch64_required_extensions::REQUIRED_EXTENSIONS }, #define DEF_SME_ZA_FUNCTION_GS(NAME, SHAPE, TYPES, GROUPS, PREDS) \ { #NAME, &functions::NAME##_za, &shapes::SHAPE, types_##TYPES, \ - groups_##GROUPS, preds_##PREDS, (REQUIRED_EXTENSIONS | AARCH64_FL_ZA_ON) }, + groups_##GROUPS, preds_##PREDS, \ + aarch64_required_extensions::REQUIRED_EXTENSIONS \ + .and_also (AARCH64_FL_ZA_ON) }, #include "aarch64-sve-builtins-sme.def" }; @@ -1417,16 +1422,17 @@ add_shared_state_attribute (const char *name, bool is_in, bool is_out, } /* Return the appropriate function attributes for INSTANCE, which requires - the feature flags in REQUIRED_EXTENSIONS. */ + the architecture extensions in REQUIRED_EXTENSIONS. */ tree function_builder::get_attributes (const function_instance &instance, - aarch64_feature_flags required_extensions) + aarch64_required_extensions + required_extensions) { tree attrs = NULL_TREE; - if (required_extensions & AARCH64_FL_SM_ON) + if (required_extensions.sm_off == 0) attrs = add_attribute ("arm", "streaming", NULL_TREE, attrs); - else if (!(required_extensions & AARCH64_FL_SM_OFF)) + else if (required_extensions.sm_on != 0) attrs = add_attribute ("arm", "streaming_compatible", NULL_TREE, attrs); attrs = add_shared_state_attribute ("in", true, false, @@ -1452,12 +1458,13 @@ function_builder::get_attributes (const function_instance &instance, /* Add a function called NAME with type FNTYPE and attributes ATTRS. INSTANCE describes what the function does and OVERLOADED_P indicates - whether it is overloaded. REQUIRED_EXTENSIONS are the set of - architecture extensions that the function requires. */ + whether it is overloaded. REQUIRED_EXTENSIONS describes the architecture + extensions that the function requires. */ registered_function & function_builder::add_function (const function_instance &instance, const char *name, tree fntype, tree attrs, - aarch64_feature_flags required_extensions, + aarch64_required_extensions + required_extensions, bool overloaded_p, bool placeholder_p) { @@ -1497,7 +1504,7 @@ function_builder::add_function (const function_instance &instance, /* Add a built-in function for INSTANCE, with the argument types given by ARGUMENT_TYPES and the return type given by RETURN_TYPE. - REQUIRED_EXTENSIONS are the set of architecture extensions that the + REQUIRED_EXTENSIONS describes the architecture extensions that the function requires. FORCE_DIRECT_OVERLOADS is true if there is a one-to-one mapping between "short" and "full" names, and if standard overload resolution therefore isn't necessary. */ @@ -1506,7 +1513,7 @@ function_builder:: add_unique_function (const function_instance &instance, tree return_type, vec &argument_types, - aarch64_feature_flags required_extensions, + aarch64_required_extensions required_extensions, bool force_direct_overloads) { /* Add the function under its full (unique) name. */ @@ -1544,7 +1551,7 @@ add_unique_function (const function_instance &instance, } /* Add one function decl for INSTANCE, to be used with manual overload - resolution. REQUIRED_EXTENSIONS are the set of architecture extensions + resolution. REQUIRED_EXTENSIONS describes the architecture extensions that the function requires. For simplicity, deal with duplicate attempts to add the same function, @@ -1555,7 +1562,7 @@ add_unique_function (const function_instance &instance, void function_builder:: add_overloaded_function (const function_instance &instance, - aarch64_feature_flags required_extensions) + aarch64_required_extensions required_extensions) { auto &name_map = overload_names[m_function_nulls]; if (!name_map) @@ -1565,8 +1572,12 @@ add_overloaded_function (const function_instance &instance, tree id = get_identifier (name); if (registered_function **map_value = name_map->get (id)) gcc_assert ((*map_value)->instance == instance - && ((*map_value)->required_extensions - & ~required_extensions) == 0); + && (required_extensions.sm_off == 0 + || ((*map_value)->required_extensions.sm_off + & ~required_extensions.sm_off) == 0) + && (required_extensions.sm_on == 0 + || ((*map_value)->required_extensions.sm_on + & ~required_extensions.sm_on) == 0)); else { registered_function &rfn diff --git a/gcc/config/aarch64/aarch64-sve-builtins.h b/gcc/config/aarch64/aarch64-sve-builtins.h index 4cdc0541bdc..d5cc6e0a40d 100644 --- a/gcc/config/aarch64/aarch64-sve-builtins.h +++ b/gcc/config/aarch64/aarch64-sve-builtins.h @@ -363,9 +363,8 @@ struct function_group_info const group_suffix_index *groups; const predication_index *preds; - /* The architecture extensions that the functions require, as a set of - AARCH64_FL_* flags. */ - aarch64_feature_flags required_extensions; + /* The architecture extensions that the functions require. */ + aarch64_required_extensions required_extensions; }; /* Describes a single fully-resolved function (i.e. one that has a @@ -432,9 +431,9 @@ public: ~function_builder (); void add_unique_function (const function_instance &, tree, - vec &, aarch64_feature_flags, bool); + vec &, aarch64_required_extensions, bool); void add_overloaded_function (const function_instance &, - aarch64_feature_flags); + aarch64_required_extensions); void add_overloaded_functions (const function_group_info &, mode_suffix_index); @@ -446,11 +445,11 @@ private: char *get_name (const function_instance &, bool); - tree get_attributes (const function_instance &, aarch64_feature_flags); + tree get_attributes (const function_instance &, aarch64_required_extensions); registered_function &add_function (const function_instance &, const char *, tree, tree, - aarch64_feature_flags, bool, bool); + aarch64_required_extensions, bool, bool); /* The function type to use for functions that are resolved by function_resolver. */