From patchwork Tue Dec 6 10:13:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arthur Cohen X-Patchwork-Id: 61508 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7C96E382CE30 for ; Tue, 6 Dec 2022 10:17:16 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by sourceware.org (Postfix) with ESMTPS id 7B99C396E856 for ; Tue, 6 Dec 2022 10:12:13 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7B99C396E856 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=embecosm.com Received: by mail-wm1-x336.google.com with SMTP id o7-20020a05600c510700b003cffc0b3374so12318693wms.0 for ; Tue, 06 Dec 2022 02:12:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=embecosm.com; s=google; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=u71FHUfeZgdiXgrB6yc4a6UfoEtlcUjpXI9cbioAdDI=; b=Vt8i/98NIX8Fzb5hu3zg9q7E6FGH3r0qrqvBz7NHX/IO84+fzCMw+zD61egJ8choCw NOGIQ2IaKr4PBe0TXJ+sP2b81Qk9qYpp5HEB2VWpQexC0ZZVwV+B90MlwIO9aCm0pxHR p5/00gvZzgXJDlTUqLey/FbGSf0Hxf5JvdHQxCsaJ3h+Yak7xgaiMMnAOon4/PaFo+9z 1xt74EQgegw/t7Z+l9TGBaFSfC6/hyQTRFg5DCtkVPN8q+IkbsdQIdB59ooSA0uwpTLq k8ec7/PRNc8Br2mPWL/jCFJyMyZiUJQGto5Agy49M6NPXm9x8ypQwAdPHD0dZqOZJn4i SUqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=u71FHUfeZgdiXgrB6yc4a6UfoEtlcUjpXI9cbioAdDI=; b=r8AcEjK1NjPbxVZC7oIIOOAl6wVHhEhXdrnWogA5MpJTxqnVnb64BypjAYrh1XyUz7 y1BSqycVjTn3zrx3pon7zQTBPSBVvTIeKvQ1DgQhrD18YEy/Bzzb6FyBAiRoKyfykRFk LZI2YlqE+ntcbcrmBt+UJ5sO2i9QUobXuy/M+93HLUmMcleCsZMkDMdDlqW41YvVujvM J0QBOkpWn7uJz3b9I5Yu4iKqq1xMGj/qzW1rFgYSJyeVmESf+qfaj0hvYtvzTsoA0yU+ JOPeChOG3jn3v9xDC0ZI1a3q1T7DQhRLIP9cGi5uiDVJu373u4KfGAaWto4VVh/3xdrM loow== X-Gm-Message-State: ANoB5pnXqyx0eHvRdXbPZzFyx1Idqq3vhvYCtRhzUY9+Pz5WT8FSTecZ hIHPD2Y4i6w0h0yeg/Jh1qMin/ThmzqRyxUIxQ== X-Google-Smtp-Source: AA0mqf4K0pi2QPb5ekanOsfiJ95yk0ycO5qOGyDgnVrSLbFl7b5cA4wOQDyyP59ZBI12bZ7ZVC+0MQ== X-Received: by 2002:a1c:ed17:0:b0:3cf:cfce:9d9e with SMTP id l23-20020a1ced17000000b003cfcfce9d9emr50840855wmh.66.1670321530448; Tue, 06 Dec 2022 02:12:10 -0800 (PST) Received: from platypus.lan ([2001:861:5e4c:3bb0:6424:328a:1734:3249]) by smtp.googlemail.com with ESMTPSA id r10-20020a05600c458a00b003cfd4a50d5asm27052699wmo.34.2022.12.06.02.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Dec 2022 02:12:10 -0800 (PST) From: arthur.cohen@embecosm.com To: gcc-patches@gcc.gnu.org Cc: gcc-rust@gcc.gnu.org, Arthur Cohen , Philip Herron , The Other Subject: [PATCH Rust front-end v4 15/46] gccrs: Add expansion pass for the Rust front-end Date: Tue, 6 Dec 2022 11:13:47 +0100 Message-Id: <20221206101417.778807-16-arthur.cohen@embecosm.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221206101417.778807-1-arthur.cohen@embecosm.com> References: <20221206101417.778807-1-arthur.cohen@embecosm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-22.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: arthur.cohen@embecosm.com Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" From: Arthur Cohen The expansion pass is responsible for two actions on our AST: 1. Expanding macro calls 2. Performing conditional compilation Calls to macros should be checked and expanded into an AST fragment based on the context they've been called in. This is similar to token substitution, with a lot of intricacies and checks being performed. A single invocation can result in an AST fragment containing multiple statements or multiple expressions, which need to be handled as well. Furthermore, Rust macros can contain repetitions relying on Kleine operators, similar to regular expression patterns, that also need to be expanded properly. Finally, Rust code can be hidden behind `cfg` directives, which allow the user to perform conditional compilation. If a `cfg` predicate is not met, the expression or statement it refers to should be marked for strip and removed from the AST. Co-authored-by: Philip Herron Co-authored-by: The Other --- gcc/rust/expand/rust-attribute-visitor.cc | 3445 ++++++++++++++++++ gcc/rust/expand/rust-attribute-visitor.h | 316 ++ gcc/rust/expand/rust-macro-builtins.cc | 484 +++ gcc/rust/expand/rust-macro-builtins.h | 107 + gcc/rust/expand/rust-macro-expand.cc | 1012 +++++ gcc/rust/expand/rust-macro-expand.h | 366 ++ gcc/rust/expand/rust-macro-invoc-lexer.cc | 29 + gcc/rust/expand/rust-macro-invoc-lexer.h | 64 + gcc/rust/expand/rust-macro-substitute-ctx.cc | 312 ++ gcc/rust/expand/rust-macro-substitute-ctx.h | 93 + 10 files changed, 6228 insertions(+) create mode 100644 gcc/rust/expand/rust-attribute-visitor.cc create mode 100644 gcc/rust/expand/rust-attribute-visitor.h create mode 100644 gcc/rust/expand/rust-macro-builtins.cc create mode 100644 gcc/rust/expand/rust-macro-builtins.h create mode 100644 gcc/rust/expand/rust-macro-expand.cc create mode 100644 gcc/rust/expand/rust-macro-expand.h create mode 100644 gcc/rust/expand/rust-macro-invoc-lexer.cc create mode 100644 gcc/rust/expand/rust-macro-invoc-lexer.h create mode 100644 gcc/rust/expand/rust-macro-substitute-ctx.cc create mode 100644 gcc/rust/expand/rust-macro-substitute-ctx.h diff --git a/gcc/rust/expand/rust-attribute-visitor.cc b/gcc/rust/expand/rust-attribute-visitor.cc new file mode 100644 index 00000000000..8016f9430eb --- /dev/null +++ b/gcc/rust/expand/rust-attribute-visitor.cc @@ -0,0 +1,3445 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#include "rust-attribute-visitor.h" +#include "rust-session-manager.h" + +namespace Rust { + +// Visitor used to expand attributes. +void +AttrVisitor::expand_struct_fields (std::vector &fields) +{ + for (auto it = fields.begin (); it != fields.end ();) + { + auto &field = *it; + + auto &field_attrs = field.get_outer_attrs (); + expander.expand_cfg_attrs (field_attrs); + if (expander.fails_cfg_with_expand (field_attrs)) + { + it = fields.erase (it); + continue; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + // expand sub-types of type, but can't strip type itself + auto &type = field.get_field_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + + // if nothing else happens, increment + ++it; + } +} + +void +AttrVisitor::expand_tuple_fields (std::vector &fields) +{ + for (auto it = fields.begin (); it != fields.end ();) + { + auto &field = *it; + + auto &field_attrs = field.get_outer_attrs (); + expander.expand_cfg_attrs (field_attrs); + if (expander.fails_cfg_with_expand (field_attrs)) + { + it = fields.erase (it); + continue; + } + + // expand sub-types of type, but can't strip type itself + auto &type = field.get_field_type (); + type->accept_vis (*this); + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + // if nothing else happens, increment + ++it; + } +} + +void +AttrVisitor::expand_function_params (std::vector ¶ms) +{ + expander.push_context (MacroExpander::ContextType::TYPE); + + for (auto it = params.begin (); it != params.end ();) + { + auto ¶m = *it; + + auto ¶m_attrs = param.get_outer_attrs (); + expander.expand_cfg_attrs (param_attrs); + if (expander.fails_cfg_with_expand (param_attrs)) + { + it = params.erase (it); + continue; + } + + // TODO: should an unwanted strip lead to break out of loop? + auto &pattern = param.get_pattern (); + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + + auto &type = param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + // increment + ++it; + } + + expander.pop_context (); +} + +void +AttrVisitor::expand_generic_args (AST::GenericArgs &args) +{ + // lifetime args can't be expanded + // FIXME: Can we have macro invocations for lifetimes? + + expander.push_context (MacroExpander::ContextType::TYPE); + + // expand type args - strip sub-types only + for (auto &arg : args.get_generic_args ()) + { + switch (arg.get_kind ()) + { + case AST::GenericArg::Kind::Type: { + auto &type = arg.get_type (); + type->accept_vis (*this); + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + break; + } + case AST::GenericArg::Kind::Const: { + auto &expr = arg.get_expression (); + expr->accept_vis (*this); + maybe_expand_expr (expr); + + if (expr->is_marked_for_strip ()) + rust_error_at (expr->get_locus (), + "cannot strip expression in this position"); + break; + } + default: + break; + // FIXME: Figure out what to do here if there is ambiguity. Since the + // resolver comes after the expansion, we need to figure out a way to + // strip ambiguous values here + // TODO: Arthur: Probably add a `mark_as_strip` method to `GenericArg` + // or something. This would clean up this whole thing + } + } + + expander.pop_context (); + + // FIXME: Can we have macro invocations in generic type bindings? + // expand binding args - strip sub-types only + for (auto &binding : args.get_binding_args ()) + { + auto &type = binding.get_type (); + type->accept_vis (*this); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + } +} + +void +AttrVisitor::expand_qualified_path_type (AST::QualifiedPathType &path_type) +{ + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = path_type.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + expander.pop_context (); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + if (path_type.has_as_clause ()) + { + auto &type_path = path_type.get_as_type_path (); + visit (type_path); + if (type_path.is_marked_for_strip ()) + rust_error_at (type_path.get_locus (), + "cannot strip type path in this position"); + } +} + +void +AttrVisitor::AttrVisitor::expand_closure_params ( + std::vector ¶ms) +{ + for (auto it = params.begin (); it != params.end ();) + { + auto ¶m = *it; + + auto ¶m_attrs = param.get_outer_attrs (); + expander.expand_cfg_attrs (param_attrs); + if (expander.fails_cfg_with_expand (param_attrs)) + { + it = params.erase (it); + continue; + } + + auto &pattern = param.get_pattern (); + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + + if (param.has_type_given ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + auto &type = param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + // increment if found nothing else so far + ++it; + } +} + +void +AttrVisitor::expand_self_param (AST::SelfParam &self_param) +{ + if (self_param.has_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + auto &type = self_param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + /* TODO: maybe check for invariants being violated - e.g. both type and + * lifetime? */ +} + +void +AttrVisitor::expand_where_clause (AST::WhereClause &where_clause) +{ + // items cannot be stripped conceptually, so just accept visitor + for (auto &item : where_clause.get_items ()) + item->accept_vis (*this); +} + +void +AttrVisitor::expand_trait_function_decl (AST::TraitFunctionDecl &decl) +{ + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : decl.get_generic_params ()) + param->accept_vis (*this); + + /* strip function parameters if required - this is specifically + * allowed by spec */ + expand_function_params (decl.get_function_params ()); + + if (decl.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = decl.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + if (decl.has_where_clause ()) + expand_where_clause (decl.get_where_clause ()); +} + +void +AttrVisitor::expand_trait_method_decl (AST::TraitMethodDecl &decl) +{ + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : decl.get_generic_params ()) + param->accept_vis (*this); + + /* assuming you can't strip self param - wouldn't be a method + * anymore. spec allows outer attrs on self param, but doesn't + * specify whether cfg is used. */ + expand_self_param (decl.get_self_param ()); + + /* strip function parameters if required - this is specifically + * allowed by spec */ + expand_function_params (decl.get_function_params ()); + + if (decl.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = decl.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + if (decl.has_where_clause ()) + expand_where_clause (decl.get_where_clause ()); +} + +void +AttrVisitor::visit (AST::Token &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::DelimTokenTree &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::AttrInputMetaItemContainer &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::IdentifierExpr &ident_expr) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (ident_expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (ident_expr.get_outer_attrs ())) + { + ident_expr.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::Lifetime &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::LifetimeParam &) +{ + // supposedly does not require - cfg does nothing +} +void +AttrVisitor::visit (AST::ConstGenericParam &) +{ + // likewise +} + +void +AttrVisitor::visit (AST::MacroInvocation ¯o_invoc) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (macro_invoc.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (macro_invoc.get_outer_attrs ())) + { + macro_invoc.mark_for_strip (); + return; + } + + // can't strip simple path + + // I don't think any macro token trees can be stripped in any way + + // TODO: maybe have cfg! macro stripping behaviour here? + expander.expand_invoc (macro_invoc, macro_invoc.has_semicolon ()); +} + +void +AttrVisitor::visit (AST::PathInExpression &path) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (path.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (path.get_outer_attrs ())) + { + path.mark_for_strip (); + return; + } + + for (auto &segment : path.get_segments ()) + { + if (segment.has_generic_args ()) + expand_generic_args (segment.get_generic_args ()); + } +} +void +AttrVisitor::visit (AST::TypePathSegment &) +{ + // shouldn't require +} +void +AttrVisitor::visit (AST::TypePathSegmentGeneric &segment) +{ + // TODO: strip inside generic args + + if (!segment.has_generic_args ()) + return; + + expand_generic_args (segment.get_generic_args ()); +} +void +AttrVisitor::visit (AST::TypePathSegmentFunction &segment) +{ + auto &type_path_function = segment.get_type_path_function (); + + for (auto &type : type_path_function.get_params ()) + { + type->accept_vis (*this); + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + } + + if (type_path_function.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = type_path_function.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } +} +void +AttrVisitor::visit (AST::TypePath &path) +{ + // this shouldn't strip any segments, but can strip inside them + for (auto &segment : path.get_segments ()) + segment->accept_vis (*this); +} +void +AttrVisitor::visit (AST::QualifiedPathInExpression &path) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (path.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (path.get_outer_attrs ())) + { + path.mark_for_strip (); + return; + } + + expand_qualified_path_type (path.get_qualified_path_type ()); + + for (auto &segment : path.get_segments ()) + { + if (segment.has_generic_args ()) + expand_generic_args (segment.get_generic_args ()); + } +} +void +AttrVisitor::visit (AST::QualifiedPathInType &path) +{ + expand_qualified_path_type (path.get_qualified_path_type ()); + + // this shouldn't strip any segments, but can strip inside them + for (auto &segment : path.get_segments ()) + segment->accept_vis (*this); +} + +void +AttrVisitor::visit (AST::LiteralExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::AttrInputLiteral &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::MetaItemLitExpr &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::MetaItemPathLit &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::BorrowExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &borrowed_expr = expr.get_borrowed_expr (); + borrowed_expr->accept_vis (*this); + if (borrowed_expr->is_marked_for_strip ()) + rust_error_at (borrowed_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::DereferenceExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &dereferenced_expr = expr.get_dereferenced_expr (); + dereferenced_expr->accept_vis (*this); + if (dereferenced_expr->is_marked_for_strip ()) + rust_error_at (dereferenced_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ErrorPropagationExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &propagating_expr = expr.get_propagating_expr (); + propagating_expr->accept_vis (*this); + if (propagating_expr->is_marked_for_strip ()) + rust_error_at (propagating_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::NegationExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &negated_expr = expr.get_negated_expr (); + negated_expr->accept_vis (*this); + if (negated_expr->is_marked_for_strip ()) + rust_error_at (negated_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ArithmeticOrLogicalExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &l_expr = expr.get_left_expr (); + l_expr->accept_vis (*this); + maybe_expand_expr (l_expr); + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &r_expr = expr.get_right_expr (); + r_expr->accept_vis (*this); + maybe_expand_expr (r_expr); + + // ensure that they are not marked for strip + if (expr.get_left_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_left_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before binary op exprs"); + if (expr.get_right_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_right_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ComparisonExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &l_expr = expr.get_left_expr (); + l_expr->accept_vis (*this); + maybe_expand_expr (l_expr); + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &r_expr = expr.get_right_expr (); + r_expr->accept_vis (*this); + maybe_expand_expr (r_expr); + + // ensure that they are not marked for strip + if (expr.get_left_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_left_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before binary op exprs"); + if (expr.get_right_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_right_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::LazyBooleanExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &l_expr = expr.get_left_expr (); + l_expr->accept_vis (*this); + maybe_expand_expr (l_expr); + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &r_expr = expr.get_right_expr (); + r_expr->accept_vis (*this); + maybe_expand_expr (r_expr); + + // ensure that they are not marked for strip + if (expr.get_left_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_left_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before binary op exprs"); + if (expr.get_right_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_right_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::TypeCastExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * direct descendant expression, can strip ones below that */ + + auto &casted_expr = expr.get_casted_expr (); + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + casted_expr->accept_vis (*this); + + // ensure that they are not marked for strip + if (casted_expr->is_marked_for_strip ()) + rust_error_at (casted_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed before cast exprs"); + + // TODO: strip sub-types of type + auto &type = expr.get_type_to_cast_to (); + type->accept_vis (*this); + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::AssignmentExpr &expr) +{ + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &l_expr = expr.get_left_expr (); + l_expr->accept_vis (*this); + maybe_expand_expr (l_expr); + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &r_expr = expr.get_right_expr (); + r_expr->accept_vis (*this); + maybe_expand_expr (r_expr); + + // ensure that they are not marked for strip + if (expr.get_left_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_left_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before binary op exprs"); + if (expr.get_right_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_right_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::CompoundAssignmentExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &l_expr = expr.get_left_expr (); + l_expr->accept_vis (*this); + maybe_expand_expr (l_expr); + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &r_expr = expr.get_right_expr (); + r_expr->accept_vis (*this); + maybe_expand_expr (r_expr); + + // ensure that they are not marked for strip + if (expr.get_left_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_left_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before binary op exprs"); + if (expr.get_right_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_right_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::GroupedExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says these are inner + * attributes, not outer attributes of inner expr */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &inner_expr = expr.get_expr_in_parens (); + inner_expr->accept_vis (*this); + if (inner_expr->is_marked_for_strip ()) + rust_error_at (inner_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ArrayElemsValues &elems) +{ + /* apparently outer attributes are allowed in "elements of array + * expressions" according to spec */ + expand_pointer_allow_strip (elems.get_values ()); +} +void +AttrVisitor::visit (AST::ArrayElemsCopied &elems) +{ + /* apparently outer attributes are allowed in "elements of array + * expressions" according to spec. on the other hand, it would not + * make conceptual sense to be able to remove either expression. As + * such, not implementing. TODO clear up the ambiguity here */ + + // only intend stripping for internal sub-expressions + auto &copied_expr = elems.get_elem_to_copy (); + copied_expr->accept_vis (*this); + if (copied_expr->is_marked_for_strip ()) + rust_error_at (copied_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + auto ©_count = elems.get_num_copies (); + copy_count->accept_vis (*this); + if (copy_count->is_marked_for_strip ()) + rust_error_at (copy_count->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ArrayExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says there are separate + * inner attributes, not just outer attributes of inner exprs */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* assuming you can't strip away the ArrayElems type, but can strip + * internal expressions and whatever */ + expr.get_array_elems ()->accept_vis (*this); +} +void +AttrVisitor::visit (AST::ArrayIndexExpr &expr) +{ + /* it is unclear whether outer attributes are supposed to be + * allowed, but conceptually it wouldn't make much sense, but + * having expansion code anyway. TODO */ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &array_expr = expr.get_array_expr (); + array_expr->accept_vis (*this); + if (array_expr->is_marked_for_strip ()) + rust_error_at (array_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + auto &index_expr = expr.get_index_expr (); + index_expr->accept_vis (*this); + if (index_expr->is_marked_for_strip ()) + rust_error_at (index_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::TupleExpr &expr) +{ + /* according to spec, outer attributes are allowed on "elements of + * tuple expressions" */ + + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says these are inner + * attributes, not outer attributes of inner expr */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* apparently outer attributes are allowed in "elements of tuple + * expressions" according to spec */ + expand_pointer_allow_strip (expr.get_tuple_elems ()); +} +void +AttrVisitor::visit (AST::TupleIndexExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* wouldn't strip this directly (as outer attrs should be + * associated with this level), but any sub-expressions would be + * stripped. Thus, no need to erase when strip check called. */ + auto &tuple_expr = expr.get_tuple_expr (); + tuple_expr->accept_vis (*this); + if (tuple_expr->is_marked_for_strip ()) + rust_error_at (tuple_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::StructExprStruct &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says these are inner + * attributes, not outer attributes of inner expr */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // strip sub-exprs of path + auto &struct_name = expr.get_struct_name (); + visit (struct_name); + if (struct_name.is_marked_for_strip ()) + rust_error_at (struct_name.get_locus (), + "cannot strip path in this position"); +} +void +AttrVisitor::visit (AST::StructExprFieldIdentifier &) +{ + // as no attrs (at moment, at least), no stripping possible +} +void +AttrVisitor::visit (AST::StructExprFieldIdentifierValue &field) +{ + /* as no attrs possible (at moment, at least), only sub-expression + * stripping is possible */ + auto &value = field.get_value (); + value->accept_vis (*this); + if (value->is_marked_for_strip ()) + rust_error_at (value->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::StructExprFieldIndexValue &field) +{ + /* as no attrs possible (at moment, at least), only sub-expression + * stripping is possible */ + auto &value = field.get_value (); + value->accept_vis (*this); + if (value->is_marked_for_strip ()) + rust_error_at (value->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::StructExprStructFields &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says these are inner + * attributes, not outer attributes of inner expr */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // strip sub-exprs of path + auto &struct_name = expr.get_struct_name (); + visit (struct_name); + if (struct_name.is_marked_for_strip ()) + rust_error_at (struct_name.get_locus (), + "cannot strip path in this position"); + + /* spec does not specify whether expressions are allowed to be + * stripped at top level of struct fields, but I wouldn't think + * that they would be, so operating under the assumption that only + * sub-expressions can be stripped. */ + for (auto &field : expr.get_fields ()) + { + field->accept_vis (*this); + // shouldn't strip in this + } + + /* struct base presumably can't be stripped, as the '..' is before + * the expression. as such, can only strip sub-expressions. */ + if (expr.has_struct_base ()) + { + auto &base_struct_expr = expr.get_struct_base ().get_base_struct (); + base_struct_expr->accept_vis (*this); + if (base_struct_expr->is_marked_for_strip ()) + rust_error_at (base_struct_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + } +} +void +AttrVisitor::visit (AST::StructExprStructBase &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says these are inner + * attributes, not outer attributes of inner expr */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // strip sub-exprs of path + auto &struct_name = expr.get_struct_name (); + visit (struct_name); + if (struct_name.is_marked_for_strip ()) + rust_error_at (struct_name.get_locus (), + "cannot strip path in this position"); + + /* struct base presumably can't be stripped, as the '..' is before + * the expression. as such, can only strip sub-expressions. */ + rust_assert (!expr.get_struct_base ().is_invalid ()); + auto &base_struct_expr = expr.get_struct_base ().get_base_struct (); + base_struct_expr->accept_vis (*this); + if (base_struct_expr->is_marked_for_strip ()) + rust_error_at (base_struct_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::CallExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* should not be outer attrs on "function" expression - outer attrs + * should be associated with call expr as a whole. only sub-expr + * expansion is possible. */ + auto &function = expr.get_function_expr (); + function->accept_vis (*this); + if (function->is_marked_for_strip ()) + rust_error_at (function->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + /* spec says outer attributes are specifically allowed for elements + * of call expressions, so full stripping possible */ + // FIXME: Arthur: Figure out how to refactor this - This is similar to + // expanding items in the crate or stmts in blocks + expand_pointer_allow_strip (expr.get_params ()); + auto ¶ms = expr.get_params (); + for (auto it = params.begin (); it != params.end ();) + { + auto &stmt = *it; + + stmt->accept_vis (*this); + + auto final_fragment = expand_macro_fragment_recursive (); + if (final_fragment.should_expand ()) + { + // Remove the current expanded invocation + it = params.erase (it); + for (auto &node : final_fragment.get_nodes ()) + { + it = params.insert (it, node.take_expr ()); + it++; + } + } + else if (stmt->is_marked_for_strip ()) + it = params.erase (it); + else + it++; + } +} +void +AttrVisitor::visit (AST::MethodCallExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* should not be outer attrs on "receiver" expression - outer attrs + * should be associated with call expr as a whole. only sub-expr + * expansion is possible. */ + auto &receiver = expr.get_receiver_expr (); + receiver->accept_vis (*this); + if (receiver->is_marked_for_strip ()) + rust_error_at (receiver->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + auto &method_name = expr.get_method_name (); + if (method_name.has_generic_args ()) + expand_generic_args (method_name.get_generic_args ()); + + /* spec says outer attributes are specifically allowed for elements + * of method call expressions, so full stripping possible */ + expand_pointer_allow_strip (expr.get_params ()); +} +void +AttrVisitor::visit (AST::FieldAccessExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* should not be outer attrs on "receiver" expression - outer attrs + * should be associated with field expr as a whole. only sub-expr + * expansion is possible. */ + auto &receiver = expr.get_receiver_expr (); + receiver->accept_vis (*this); + if (receiver->is_marked_for_strip ()) + rust_error_at (receiver->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ClosureExprInner &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip closure parameters if required - this is specifically + * allowed by spec */ + expand_closure_params (expr.get_params ()); + + // can't strip expression itself, but can strip sub-expressions + auto &definition_expr = expr.get_definition_expr (); + definition_expr->accept_vis (*this); + if (definition_expr->is_marked_for_strip ()) + rust_error_at (definition_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} + +void +AttrVisitor::visit (AST::BlockExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip test based on inner attrs - spec says there are inner + * attributes, not just outer attributes of inner stmts */ + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + std::function (AST::SingleASTNode)> extractor + = [] (AST::SingleASTNode node) { return node.take_stmt (); }; + + expand_macro_children (MacroExpander::BLOCK, expr.get_statements (), + extractor); + + expander.push_context (MacroExpander::BLOCK); + + // strip tail expression if exists - can actually fully remove it + if (expr.has_tail_expr ()) + { + auto &tail_expr = expr.get_tail_expr (); + + tail_expr->accept_vis (*this); + maybe_expand_expr (tail_expr); + + if (tail_expr->is_marked_for_strip ()) + expr.strip_tail_expr (); + } + expander.pop_context (); +} + +void +AttrVisitor::visit (AST::ClosureExprInnerTyped &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* strip closure parameters if required - this is specifically + * allowed by spec */ + expand_closure_params (expr.get_params ()); + + expander.push_context (MacroExpander::ContextType::TYPE); + + // can't strip return type, but can strip sub-types + auto &type = expr.get_return_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + // can't strip expression itself, but can strip sub-expressions + auto &definition_block = expr.get_definition_block (); + definition_block->accept_vis (*this); + if (definition_block->is_marked_for_strip ()) + rust_error_at (definition_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ContinueExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::BreakExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* spec does not say that you can have outer attributes on + * expression, so assuming you can't. stripping for sub-expressions + * is the only thing that can be done */ + if (expr.has_break_expr ()) + { + auto &break_expr = expr.get_break_expr (); + + break_expr->accept_vis (*this); + + if (break_expr->is_marked_for_strip ()) + rust_error_at (break_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + } +} +void +AttrVisitor::visit (AST::RangeFromToExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + expr.get_from_expr ()->accept_vis (*this); + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + expr.get_to_expr ()->accept_vis (*this); + + // ensure that they are not marked for strip + if (expr.get_from_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_from_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before range exprs"); + if (expr.get_to_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_to_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::RangeFromExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * direct descendant expression, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + auto &from_expr = expr.get_from_expr (); + + from_expr->accept_vis (*this); + + if (from_expr->is_marked_for_strip ()) + rust_error_at (from_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed before range exprs"); +} +void +AttrVisitor::visit (AST::RangeToExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * direct descendant expression, can strip ones below that */ + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &to_expr = expr.get_to_expr (); + + to_expr->accept_vis (*this); + + if (to_expr->is_marked_for_strip ()) + rust_error_at (to_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::RangeFullExpr &) +{ + // outer attributes never allowed before these, so no stripping +} +void +AttrVisitor::visit (AST::RangeFromToInclExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * two direct descendant expressions, can strip ones below that */ + + /* should have no possibility for outer attrs as would be parsed + * with outer expr */ + expr.get_from_expr ()->accept_vis (*this); + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + expr.get_to_expr ()->accept_vis (*this); + + // ensure that they are not marked for strip + if (expr.get_from_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_from_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes are never allowed " + "before range exprs"); + if (expr.get_to_expr ()->is_marked_for_strip ()) + rust_error_at (expr.get_to_expr ()->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::RangeToInclExpr &expr) +{ + /* outer attributes never allowed before these. while cannot strip + * direct descendant expression, can strip ones below that */ + + /* should syntactically not have outer attributes, though this may + * not have worked in practice */ + auto &to_expr = expr.get_to_expr (); + + to_expr->accept_vis (*this); + + if (to_expr->is_marked_for_strip ()) + rust_error_at (to_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ReturnExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* spec does not say that you can have outer attributes on + * expression, so assuming you can't. stripping for sub-expressions + * is the only thing that can be done */ + if (expr.has_returned_expr ()) + { + auto &returned_expr = expr.get_returned_expr (); + + returned_expr->accept_vis (*this); + + if (returned_expr->is_marked_for_strip ()) + rust_error_at (returned_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + } + /* TODO: conceptually, you would maybe be able to remove a returned + * expr - e.g. if you had conditional compilation returning void or + * returning a type. On the other hand, I think that function + * return type cannot be conditionally compiled, so I assumed you + * can't do this either. */ +} +void +AttrVisitor::visit (AST::UnsafeBlockExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip block itself, but can strip sub-expressions + auto &block_expr = expr.get_block_expr (); + block_expr->accept_vis (*this); + if (block_expr->is_marked_for_strip ()) + rust_error_at (block_expr->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::LoopExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip block itself, but can strip sub-expressions + auto &loop_block = expr.get_loop_block (); + loop_block->accept_vis (*this); + if (loop_block->is_marked_for_strip ()) + rust_error_at (loop_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::WhileLoopExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip predicate expr itself, but can strip sub-expressions + auto &predicate_expr = expr.get_predicate_expr (); + predicate_expr->accept_vis (*this); + if (predicate_expr->is_marked_for_strip ()) + rust_error_at (predicate_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip block itself, but can strip sub-expressions + auto &loop_block = expr.get_loop_block (); + loop_block->accept_vis (*this); + if (loop_block->is_marked_for_strip ()) + rust_error_at (loop_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::WhileLetLoopExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + for (auto &pattern : expr.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + // can't strip scrutinee expr itself, but can strip sub-expressions + auto &scrutinee_expr = expr.get_scrutinee_expr (); + scrutinee_expr->accept_vis (*this); + if (scrutinee_expr->is_marked_for_strip ()) + rust_error_at (scrutinee_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip block itself, but can strip sub-expressions + auto &loop_block = expr.get_loop_block (); + loop_block->accept_vis (*this); + if (loop_block->is_marked_for_strip ()) + rust_error_at (loop_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::ForLoopExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // strip sub-patterns of pattern + auto &pattern = expr.get_pattern (); + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + + // can't strip scrutinee expr itself, but can strip sub-expressions + auto &iterator_expr = expr.get_iterator_expr (); + iterator_expr->accept_vis (*this); + if (iterator_expr->is_marked_for_strip ()) + rust_error_at (iterator_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip block itself, but can strip sub-expressions + auto &loop_block = expr.get_loop_block (); + loop_block->accept_vis (*this); + if (loop_block->is_marked_for_strip ()) + rust_error_at (loop_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfExpr &expr) +{ + // rust playground test shows that IfExpr does support outer attrs, at least + // when used as statement + + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip condition expr itself, but can strip sub-expressions + auto &condition_expr = expr.get_condition_expr (); + condition_expr->accept_vis (*this); + maybe_expand_expr (condition_expr); + if (condition_expr->is_marked_for_strip ()) + rust_error_at (condition_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfExprConseqElse &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip condition expr itself, but can strip sub-expressions + auto &condition_expr = expr.get_condition_expr (); + condition_expr->accept_vis (*this); + maybe_expand_expr (condition_expr); + if (condition_expr->is_marked_for_strip ()) + rust_error_at (condition_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip else block itself, but can strip sub-expressions + auto &else_block = expr.get_else_block (); + else_block->accept_vis (*this); + if (else_block->is_marked_for_strip ()) + rust_error_at (else_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfExprConseqIf &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip condition expr itself, but can strip sub-expressions + auto &condition_expr = expr.get_condition_expr (); + condition_expr->accept_vis (*this); + maybe_expand_expr (condition_expr); + if (condition_expr->is_marked_for_strip ()) + rust_error_at (condition_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip if expr itself, but can strip sub-expressions + auto &conseq_if_expr = expr.get_conseq_if_expr (); + conseq_if_expr->accept_vis (*this); + if (conseq_if_expr->is_marked_for_strip ()) + rust_error_at (conseq_if_expr->get_locus (), + "cannot strip consequent if expression in this " + "position - outer attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfExprConseqIfLet &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip condition expr itself, but can strip sub-expressions + auto &condition_expr = expr.get_condition_expr (); + condition_expr->accept_vis (*this); + maybe_expand_expr (condition_expr); + if (condition_expr->is_marked_for_strip ()) + rust_error_at (condition_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip if let expr itself, but can strip sub-expressions + auto &conseq_if_let_expr = expr.get_conseq_if_let_expr (); + conseq_if_let_expr->accept_vis (*this); + if (conseq_if_let_expr->is_marked_for_strip ()) + rust_error_at (conseq_if_let_expr->get_locus (), + "cannot strip consequent if let expression in this " + "position - outer attributes not " + "allowed"); +} +void +AttrVisitor::visit (AST::IfLetExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + for (auto &pattern : expr.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + // can't strip value expr itself, but can strip sub-expressions + auto &value_expr = expr.get_value_expr (); + value_expr->accept_vis (*this); + if (value_expr->is_marked_for_strip ()) + rust_error_at (value_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfLetExprConseqElse &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + for (auto &pattern : expr.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + // can't strip value expr itself, but can strip sub-expressions + auto &value_expr = expr.get_value_expr (); + value_expr->accept_vis (*this); + if (value_expr->is_marked_for_strip ()) + rust_error_at (value_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip else block itself, but can strip sub-expressions + auto &else_block = expr.get_else_block (); + else_block->accept_vis (*this); + if (else_block->is_marked_for_strip ()) + rust_error_at (else_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfLetExprConseqIf &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + for (auto &pattern : expr.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + // can't strip value expr itself, but can strip sub-expressions + auto &value_expr = expr.get_value_expr (); + value_expr->accept_vis (*this); + if (value_expr->is_marked_for_strip ()) + rust_error_at (value_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip if expr itself, but can strip sub-expressions + auto &conseq_if_expr = expr.get_conseq_if_expr (); + conseq_if_expr->accept_vis (*this); + if (conseq_if_expr->is_marked_for_strip ()) + rust_error_at (conseq_if_expr->get_locus (), + "cannot strip consequent if expression in this " + "position - outer attributes not allowed"); +} +void +AttrVisitor::visit (AST::IfLetExprConseqIfLet &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + for (auto &pattern : expr.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + // can't strip value expr itself, but can strip sub-expressions + auto &value_expr = expr.get_value_expr (); + value_expr->accept_vis (*this); + if (value_expr->is_marked_for_strip ()) + rust_error_at (value_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // can't strip if block itself, but can strip sub-expressions + auto &if_block = expr.get_if_block (); + if_block->accept_vis (*this); + if (if_block->is_marked_for_strip ()) + rust_error_at (if_block->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); + + // can't strip if let expr itself, but can strip sub-expressions + auto &conseq_if_let_expr = expr.get_conseq_if_let_expr (); + conseq_if_let_expr->accept_vis (*this); + if (conseq_if_let_expr->is_marked_for_strip ()) + rust_error_at (conseq_if_let_expr->get_locus (), + "cannot strip consequent if let expression in this " + "position - outer attributes not " + "allowed"); +} +void +AttrVisitor::visit (AST::MatchExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // inner attr strip test + expander.expand_cfg_attrs (expr.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_inner_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip scrutinee expr itself, but can strip sub-expressions + auto &scrutinee_expr = expr.get_scrutinee_expr (); + scrutinee_expr->accept_vis (*this); + if (scrutinee_expr->is_marked_for_strip ()) + rust_error_at (scrutinee_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // strip match cases + auto &match_cases = expr.get_match_cases (); + for (auto it = match_cases.begin (); it != match_cases.end ();) + { + auto &match_case = *it; + + // strip match case based on outer attributes in match arm + auto &match_arm = match_case.get_arm (); + expander.expand_cfg_attrs (match_arm.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (match_arm.get_outer_attrs ())) + { + // strip match case + it = match_cases.erase (it); + continue; + } + + for (auto &pattern : match_arm.get_patterns ()) + { + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + } + + /* assuming that guard expression cannot be stripped as + * strictly speaking you would have to strip the whole guard to + * make syntactical sense, which you can't do. as such, only + * strip sub-expressions */ + if (match_arm.has_match_arm_guard ()) + { + auto &guard_expr = match_arm.get_guard_expr (); + guard_expr->accept_vis (*this); + if (guard_expr->is_marked_for_strip ()) + rust_error_at (guard_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + } + + // strip sub-expressions from match cases + auto &case_expr = match_case.get_expr (); + case_expr->accept_vis (*this); + if (case_expr->is_marked_for_strip ()) + rust_error_at (case_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + // increment to next case if haven't continued + ++it; + } +} +void +AttrVisitor::visit (AST::AwaitExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + /* can't strip awaited expr itself, but can strip sub-expressions + * - this is because you can't have no expr to await */ + auto &awaited_expr = expr.get_awaited_expr (); + awaited_expr->accept_vis (*this); + if (awaited_expr->is_marked_for_strip ()) + rust_error_at (awaited_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::AsyncBlockExpr &expr) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (expr.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (expr.get_outer_attrs ())) + { + expr.mark_for_strip (); + return; + } + + // can't strip block itself, but can strip sub-expressions + auto &block_expr = expr.get_block_expr (); + block_expr->accept_vis (*this); + if (block_expr->is_marked_for_strip ()) + rust_error_at (block_expr->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} + +void +AttrVisitor::visit (AST::TypeParam ¶m) +{ + // outer attributes don't actually do anything, so ignore them + + if (param.has_type_param_bounds ()) + { + // don't strip directly, only components of bounds + for (auto &bound : param.get_type_param_bounds ()) + bound->accept_vis (*this); + } + + if (param.has_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + auto &type = param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } +} +void +AttrVisitor::visit (AST::LifetimeWhereClauseItem &) +{ + // shouldn't require +} +void +AttrVisitor::visit (AST::TypeBoundWhereClauseItem &item) +{ + // for lifetimes shouldn't require + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = item.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + // don't strip directly, only components of bounds + for (auto &bound : item.get_type_param_bounds ()) + bound->accept_vis (*this); +} +void +AttrVisitor::visit (AST::Method &method) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (method.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (method.get_outer_attrs ())) + { + method.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : method.get_generic_params ()) + param->accept_vis (*this); + + /* assuming you can't strip self param - wouldn't be a method + * anymore. spec allows outer attrs on self param, but doesn't + * specify whether cfg is used. */ + expand_self_param (method.get_self_param ()); + + /* strip method parameters if required - this is specifically + * allowed by spec */ + expand_function_params (method.get_function_params ()); + + if (method.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = method.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + if (method.has_where_clause ()) + expand_where_clause (method.get_where_clause ()); + + /* body should always exist - if error state, should have returned + * before now */ + // can't strip block itself, but can strip sub-expressions + auto &block_expr = method.get_definition (); + block_expr->accept_vis (*this); + if (block_expr->is_marked_for_strip ()) + rust_error_at (block_expr->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::Module &module) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (module.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (module.get_outer_attrs ())) + { + module.mark_for_strip (); + return; + } + + // A loaded module might have inner attributes + if (module.get_kind () == AST::Module::ModuleKind::LOADED) + { + // strip test based on inner attrs + expander.expand_cfg_attrs (module.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (module.get_inner_attrs ())) + { + module.mark_for_strip (); + return; + } + } + + // Parse the module's items if they haven't been expanded and the file + // should be parsed (i.e isn't hidden behind an untrue or impossible cfg + // directive) + if (!module.is_marked_for_strip () + && module.get_kind () == AST::Module::ModuleKind::UNLOADED) + { + module.load_items (); + } + + // strip items if required + expand_pointer_allow_strip (module.get_items ()); +} +void +AttrVisitor::visit (AST::ExternCrate &extern_crate) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (extern_crate.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (extern_crate.get_outer_attrs ())) + { + extern_crate.mark_for_strip (); + return; + } + + if (!extern_crate.references_self ()) + { + Session &session = Session::get_instance (); + session.load_extern_crate (extern_crate.get_referenced_crate (), + extern_crate.get_locus ()); + } +} +void +AttrVisitor::visit (AST::UseTreeGlob &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::UseTreeList &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::UseTreeRebind &) +{ + // shouldn't require? +} +void +AttrVisitor::visit (AST::UseDeclaration &use_decl) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (use_decl.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (use_decl.get_outer_attrs ())) + { + use_decl.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::Function &function) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (function.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (function.get_outer_attrs ())) + { + function.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : function.get_generic_params ()) + param->accept_vis (*this); + + /* strip function parameters if required - this is specifically + * allowed by spec */ + expand_function_params (function.get_function_params ()); + + if (function.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = function.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + if (function.has_where_clause ()) + expand_where_clause (function.get_where_clause ()); + + /* body should always exist - if error state, should have returned + * before now */ + // can't strip block itself, but can strip sub-expressions + auto &block_expr = function.get_definition (); + block_expr->accept_vis (*this); + if (block_expr->is_marked_for_strip ()) + rust_error_at (block_expr->get_locus (), + "cannot strip block expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::TypeAlias &type_alias) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (type_alias.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (type_alias.get_outer_attrs ())) + { + type_alias.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : type_alias.get_generic_params ()) + param->accept_vis (*this); + + if (type_alias.has_where_clause ()) + expand_where_clause (type_alias.get_where_clause ()); + + auto &type = type_alias.get_type_aliased (); + type->accept_vis (*this); + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::StructStruct &struct_item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (struct_item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (struct_item.get_outer_attrs ())) + { + struct_item.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : struct_item.get_generic_params ()) + param->accept_vis (*this); + + if (struct_item.has_where_clause ()) + expand_where_clause (struct_item.get_where_clause ()); + + /* strip struct fields if required - this is presumably + * allowed by spec */ + expand_struct_fields (struct_item.get_fields ()); +} +void +AttrVisitor::visit (AST::TupleStruct &tuple_struct) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (tuple_struct.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (tuple_struct.get_outer_attrs ())) + { + tuple_struct.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : tuple_struct.get_generic_params ()) + param->accept_vis (*this); + + /* strip struct fields if required - this is presumably + * allowed by spec */ + expand_tuple_fields (tuple_struct.get_fields ()); + + if (tuple_struct.has_where_clause ()) + expand_where_clause (tuple_struct.get_where_clause ()); +} +void +AttrVisitor::visit (AST::EnumItem &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::EnumItemTuple &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + /* strip item fields if required - this is presumably + * allowed by spec */ + expand_tuple_fields (item.get_tuple_fields ()); +} +void +AttrVisitor::visit (AST::EnumItemStruct &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + /* strip item fields if required - this is presumably + * allowed by spec */ + expand_struct_fields (item.get_struct_fields ()); +} +void +AttrVisitor::visit (AST::EnumItemDiscriminant &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &expr = item.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + rust_error_at (expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::Enum &enum_item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (enum_item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (enum_item.get_outer_attrs ())) + { + enum_item.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : enum_item.get_generic_params ()) + param->accept_vis (*this); + + if (enum_item.has_where_clause ()) + expand_where_clause (enum_item.get_where_clause ()); + + /* strip enum fields if required - this is presumably + * allowed by spec */ + expand_pointer_allow_strip (enum_item.get_variants ()); +} +void +AttrVisitor::visit (AST::Union &union_item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (union_item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (union_item.get_outer_attrs ())) + { + union_item.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : union_item.get_generic_params ()) + param->accept_vis (*this); + + if (union_item.has_where_clause ()) + expand_where_clause (union_item.get_where_clause ()); + + /* strip union fields if required - this is presumably + * allowed by spec */ + expand_struct_fields (union_item.get_variants ()); +} +void +AttrVisitor::visit (AST::ConstantItem &const_item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (const_item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (const_item.get_outer_attrs ())) + { + const_item.mark_for_strip (); + return; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + // strip any sub-types + auto &type = const_item.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &expr = const_item.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + rust_error_at (expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::StaticItem &static_item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (static_item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (static_item.get_outer_attrs ())) + { + static_item.mark_for_strip (); + return; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + // strip any sub-types + auto &type = static_item.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &expr = static_item.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + rust_error_at (expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); +} +void +AttrVisitor::visit (AST::TraitItemFunc &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + expand_trait_function_decl (item.get_trait_function_decl ()); + + if (item.has_definition ()) + { + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &block = item.get_definition (); + block->accept_vis (*this); + if (block->is_marked_for_strip ()) + rust_error_at (block->get_locus (), + "cannot strip block expression in this " + "position - outer attributes not allowed"); + } +} +void +AttrVisitor::visit (AST::TraitItemMethod &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + expand_trait_method_decl (item.get_trait_method_decl ()); + + if (item.has_definition ()) + { + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped. */ + auto &block = item.get_definition (); + block->accept_vis (*this); + if (block->is_marked_for_strip ()) + rust_error_at (block->get_locus (), + "cannot strip block expression in this " + "position - outer attributes not allowed"); + } +} +void +AttrVisitor::visit (AST::TraitItemConst &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + // strip any sub-types + auto &type = item.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped */ + if (item.has_expression ()) + { + auto &expr = item.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + rust_error_at (expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + } +} +void +AttrVisitor::visit (AST::TraitItemType &item) +{ + // initial test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + if (item.has_type_param_bounds ()) + { + // don't strip directly, only components of bounds + for (auto &bound : item.get_type_param_bounds ()) + bound->accept_vis (*this); + } +} +void +AttrVisitor::visit (AST::Trait &trait) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (trait.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (trait.get_outer_attrs ())) + { + trait.mark_for_strip (); + return; + } + + // strip test based on inner attrs + expander.expand_cfg_attrs (trait.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (trait.get_inner_attrs ())) + { + trait.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : trait.get_generic_params ()) + param->accept_vis (*this); + + if (trait.has_type_param_bounds ()) + { + // don't strip directly, only components of bounds + for (auto &bound : trait.get_type_param_bounds ()) + bound->accept_vis (*this); + } + + if (trait.has_where_clause ()) + expand_where_clause (trait.get_where_clause ()); + + std::function (AST::SingleASTNode)> extractor + = [] (AST::SingleASTNode node) { return node.take_trait_item (); }; + + expand_macro_children (MacroExpander::TRAIT, trait.get_trait_items (), + extractor); +} +void +AttrVisitor::visit (AST::InherentImpl &impl) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (impl.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (impl.get_outer_attrs ())) + { + impl.mark_for_strip (); + return; + } + + // strip test based on inner attrs + expander.expand_cfg_attrs (impl.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (impl.get_inner_attrs ())) + { + impl.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : impl.get_generic_params ()) + param->accept_vis (*this); + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = impl.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + if (impl.has_where_clause ()) + expand_where_clause (impl.get_where_clause ()); + + std::function (AST::SingleASTNode)> + extractor = [] (AST::SingleASTNode node) { return node.take_impl_item (); }; + + expand_macro_children (MacroExpander::IMPL, impl.get_impl_items (), + extractor); +} +void +AttrVisitor::visit (AST::TraitImpl &impl) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (impl.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (impl.get_outer_attrs ())) + { + impl.mark_for_strip (); + return; + } + + // strip test based on inner attrs + expander.expand_cfg_attrs (impl.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (impl.get_inner_attrs ())) + { + impl.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : impl.get_generic_params ()) + param->accept_vis (*this); + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = impl.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); + + auto &trait_path = impl.get_trait_path (); + visit (trait_path); + if (trait_path.is_marked_for_strip ()) + rust_error_at (trait_path.get_locus (), + "cannot strip typepath in this position"); + + if (impl.has_where_clause ()) + expand_where_clause (impl.get_where_clause ()); + + std::function (AST::SingleASTNode)> + extractor + = [] (AST::SingleASTNode node) { return node.take_trait_impl_item (); }; + + expand_macro_children (MacroExpander::TRAIT_IMPL, impl.get_impl_items (), + extractor); +} +void +AttrVisitor::visit (AST::ExternalStaticItem &item) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = item.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), "cannot strip type in this position"); + + expander.pop_context (); +} +void +AttrVisitor::visit (AST::ExternalFunctionItem &item) +{ + // strip test based on outer attrs + expander.expand_cfg_attrs (item.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (item.get_outer_attrs ())) + { + item.mark_for_strip (); + return; + } + + // just expand sub-stuff - can't actually strip generic params themselves + for (auto ¶m : item.get_generic_params ()) + param->accept_vis (*this); + + /* strip function parameters if required - this is specifically + * allowed by spec */ + auto ¶ms = item.get_function_params (); + for (auto it = params.begin (); it != params.end ();) + { + auto ¶m = *it; + + auto ¶m_attrs = param.get_outer_attrs (); + expander.expand_cfg_attrs (param_attrs); + if (expander.fails_cfg_with_expand (param_attrs)) + { + it = params.erase (it); + continue; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + + // increment if nothing else happens + ++it; + } + /* NOTE: these are extern function params, which may have different + * rules and restrictions to "normal" function params. So expansion + * handled separately. */ + + /* TODO: assuming that variadic nature cannot be stripped. If this + * is not true, then have code here to do so. */ + + if (item.has_return_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &return_type = item.get_return_type (); + return_type->accept_vis (*this); + + maybe_expand_type (return_type); + + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + if (item.has_where_clause ()) + expand_where_clause (item.get_where_clause ()); +} + +void +AttrVisitor::visit (AST::ExternBlock &block) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (block.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (block.get_outer_attrs ())) + { + block.mark_for_strip (); + return; + } + + // strip test based on inner attrs + expander.expand_cfg_attrs (block.get_inner_attrs ()); + if (expander.fails_cfg_with_expand (block.get_inner_attrs ())) + { + block.mark_for_strip (); + return; + } + + std::function (AST::SingleASTNode)> + extractor + = [] (AST::SingleASTNode node) { return node.take_external_item (); }; + + expand_macro_children (MacroExpander::EXTERN, block.get_extern_items (), + extractor); +} + +// I don't think it would be possible to strip macros without expansion +void +AttrVisitor::visit (AST::MacroMatchFragment &) +{} +void +AttrVisitor::visit (AST::MacroMatchRepetition &) +{} +void +AttrVisitor::visit (AST::MacroMatcher &) +{} +void +AttrVisitor::visit (AST::MacroRulesDefinition &rules_def) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (rules_def.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (rules_def.get_outer_attrs ())) + { + rules_def.mark_for_strip (); + return; + } + + // I don't think any macro rules can be stripped in any way + + auto path = Resolver::CanonicalPath::new_seg (rules_def.get_node_id (), + rules_def.get_rule_name ()); + expander.resolver->get_macro_scope ().insert (path, rules_def.get_node_id (), + rules_def.get_locus ()); + expander.mappings->insert_macro_def (&rules_def); + rust_debug_loc (rules_def.get_locus (), "inserting macro def: [%s]", + path.get ().c_str ()); +} + +void +AttrVisitor::visit (AST::MetaItemPath &) +{} +void +AttrVisitor::visit (AST::MetaItemSeq &) +{} +void +AttrVisitor::visit (AST::MetaWord &) +{} +void +AttrVisitor::visit (AST::MetaNameValueStr &) +{} +void +AttrVisitor::visit (AST::MetaListPaths &) +{} +void +AttrVisitor::visit (AST::MetaListNameValueStr &) +{} + +void +AttrVisitor::visit (AST::LiteralPattern &) +{ + // not possible +} +void +AttrVisitor::visit (AST::IdentifierPattern &pattern) +{ + // can only strip sub-patterns of the inner pattern to bind + if (!pattern.has_pattern_to_bind ()) + return; + + auto &sub_pattern = pattern.get_pattern_to_bind (); + sub_pattern->accept_vis (*this); + if (sub_pattern->is_marked_for_strip ()) + rust_error_at (sub_pattern->get_locus (), + "cannot strip pattern in this position"); +} +void +AttrVisitor::visit (AST::WildcardPattern &) +{ + // not possible +} +void +AttrVisitor::visit (AST::RangePatternBoundLiteral &) +{ + // not possible +} +void +AttrVisitor::visit (AST::RangePatternBoundPath &bound) +{ + // can expand path, but not strip it directly + auto &path = bound.get_path (); + visit (path); + if (path.is_marked_for_strip ()) + rust_error_at (path.get_locus (), "cannot strip path in this position"); +} +void +AttrVisitor::visit (AST::RangePatternBoundQualPath &bound) +{ + // can expand path, but not strip it directly + auto &path = bound.get_qualified_path (); + visit (path); + if (path.is_marked_for_strip ()) + rust_error_at (path.get_locus (), "cannot strip path in this position"); +} +void +AttrVisitor::visit (AST::RangePattern &pattern) +{ + // should have no capability to strip lower or upper bounds, only expand + pattern.get_lower_bound ()->accept_vis (*this); + pattern.get_upper_bound ()->accept_vis (*this); +} +void +AttrVisitor::visit (AST::ReferencePattern &pattern) +{ + auto &sub_pattern = pattern.get_referenced_pattern (); + sub_pattern->accept_vis (*this); + if (sub_pattern->is_marked_for_strip ()) + rust_error_at (sub_pattern->get_locus (), + "cannot strip pattern in this position"); +} +void +AttrVisitor::visit (AST::StructPatternFieldTuplePat &field) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (field.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (field.get_outer_attrs ())) + { + field.mark_for_strip (); + return; + } + + // strip sub-patterns (can't strip top-level pattern) + auto &sub_pattern = field.get_index_pattern (); + sub_pattern->accept_vis (*this); + if (sub_pattern->is_marked_for_strip ()) + rust_error_at (sub_pattern->get_locus (), + "cannot strip pattern in this position"); +} +void +AttrVisitor::visit (AST::StructPatternFieldIdentPat &field) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (field.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (field.get_outer_attrs ())) + { + field.mark_for_strip (); + return; + } + + // strip sub-patterns (can't strip top-level pattern) + auto &sub_pattern = field.get_ident_pattern (); + sub_pattern->accept_vis (*this); + if (sub_pattern->is_marked_for_strip ()) + rust_error_at (sub_pattern->get_locus (), + "cannot strip pattern in this position"); +} +void +AttrVisitor::visit (AST::StructPatternFieldIdent &field) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (field.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (field.get_outer_attrs ())) + { + field.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::StructPattern &pattern) +{ + // expand (but don't strip) path + auto &path = pattern.get_path (); + visit (path); + if (path.is_marked_for_strip ()) + rust_error_at (path.get_locus (), "cannot strip path in this position"); + + /* TODO: apparently struct pattern fields can have outer attrs. so can they + * be stripped? */ + if (!pattern.has_struct_pattern_elems ()) + return; + + auto &elems = pattern.get_struct_pattern_elems (); + + // assuming you can strip struct pattern fields + expand_pointer_allow_strip (elems.get_struct_pattern_fields ()); + + // assuming you can strip the ".." part + if (elems.has_etc ()) + { + expander.expand_cfg_attrs (elems.get_etc_outer_attrs ()); + if (expander.fails_cfg_with_expand (elems.get_etc_outer_attrs ())) + elems.strip_etc (); + } +} +void +AttrVisitor::visit (AST::TupleStructItemsNoRange &tuple_items) +{ + // can't strip individual patterns, only sub-patterns + for (auto &pattern : tuple_items.get_patterns ()) + { + pattern->accept_vis (*this); + + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } +} +void +AttrVisitor::visit (AST::TupleStructItemsRange &tuple_items) +{ + // can't strip individual patterns, only sub-patterns + for (auto &lower_pattern : tuple_items.get_lower_patterns ()) + { + lower_pattern->accept_vis (*this); + + if (lower_pattern->is_marked_for_strip ()) + rust_error_at (lower_pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } + for (auto &upper_pattern : tuple_items.get_upper_patterns ()) + { + upper_pattern->accept_vis (*this); + + if (upper_pattern->is_marked_for_strip ()) + rust_error_at (upper_pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } +} +void +AttrVisitor::visit (AST::TupleStructPattern &pattern) +{ + // expand (but don't strip) path + auto &path = pattern.get_path (); + visit (path); + if (path.is_marked_for_strip ()) + rust_error_at (path.get_locus (), "cannot strip path in this position"); + + if (pattern.has_items ()) + pattern.get_items ()->accept_vis (*this); +} +void +AttrVisitor::visit (AST::TuplePatternItemsMultiple &tuple_items) +{ + // can't strip individual patterns, only sub-patterns + for (auto &pattern : tuple_items.get_patterns ()) + { + pattern->accept_vis (*this); + + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } +} +void +AttrVisitor::visit (AST::TuplePatternItemsRanged &tuple_items) +{ + // can't strip individual patterns, only sub-patterns + for (auto &lower_pattern : tuple_items.get_lower_patterns ()) + { + lower_pattern->accept_vis (*this); + + if (lower_pattern->is_marked_for_strip ()) + rust_error_at (lower_pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } + for (auto &upper_pattern : tuple_items.get_upper_patterns ()) + { + upper_pattern->accept_vis (*this); + + if (upper_pattern->is_marked_for_strip ()) + rust_error_at (upper_pattern->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } +} +void +AttrVisitor::visit (AST::TuplePattern &pattern) +{ + if (pattern.has_tuple_pattern_items ()) + pattern.get_items ()->accept_vis (*this); +} +void +AttrVisitor::visit (AST::GroupedPattern &pattern) +{ + // can't strip inner pattern, only sub-patterns + auto &pattern_in_parens = pattern.get_pattern_in_parens (); + + pattern_in_parens->accept_vis (*this); + + if (pattern_in_parens->is_marked_for_strip ()) + rust_error_at (pattern_in_parens->get_locus (), + "cannot strip pattern in this position"); +} +void +AttrVisitor::visit (AST::SlicePattern &pattern) +{ + // can't strip individual patterns, only sub-patterns + for (auto &item : pattern.get_items ()) + { + item->accept_vis (*this); + + if (item->is_marked_for_strip ()) + rust_error_at (item->get_locus (), + "cannot strip pattern in this position"); + // TODO: quit stripping now? or keep going? + } +} + +void +AttrVisitor::visit (AST::EmptyStmt &) +{ + // assuming no outer attributes, so nothing can happen +} +void +AttrVisitor::visit (AST::LetStmt &stmt) +{ + // initial strip test based on outer attrs + expander.expand_cfg_attrs (stmt.get_outer_attrs ()); + if (expander.fails_cfg_with_expand (stmt.get_outer_attrs ())) + { + stmt.mark_for_strip (); + return; + } + + // can't strip pattern, but call for sub-patterns + auto &pattern = stmt.get_pattern (); + pattern->accept_vis (*this); + if (pattern->is_marked_for_strip ()) + rust_error_at (pattern->get_locus (), + "cannot strip pattern in this position"); + + // similar for type + if (stmt.has_type ()) + { + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = stmt.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + } + + /* strip any internal sub-expressions - expression itself isn't + * allowed to have external attributes in this position so can't be + * stripped */ + if (stmt.has_init_expr ()) + { + auto &init_expr = stmt.get_init_expr (); + init_expr->accept_vis (*this); + + if (init_expr->is_marked_for_strip ()) + rust_error_at (init_expr->get_locus (), + "cannot strip expression in this position - outer " + "attributes not allowed"); + + maybe_expand_expr (init_expr); + } +} +void +AttrVisitor::visit (AST::ExprStmtWithoutBlock &stmt) +{ + // outer attributes associated with expr, so rely on expr + + // guard - should prevent null pointer expr + if (stmt.is_marked_for_strip ()) + return; + + // strip if expr is to be stripped + auto &expr = stmt.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + { + stmt.mark_for_strip (); + return; + } +} +void +AttrVisitor::visit (AST::ExprStmtWithBlock &stmt) +{ + // outer attributes associated with expr, so rely on expr + + // guard - should prevent null pointer expr + if (stmt.is_marked_for_strip ()) + return; + + // strip if expr is to be stripped + auto &expr = stmt.get_expr (); + expr->accept_vis (*this); + if (expr->is_marked_for_strip ()) + { + stmt.mark_for_strip (); + return; + } +} + +void +AttrVisitor::visit (AST::TraitBound &bound) +{ + // nothing in for lifetimes to strip + + // expand but don't strip type path + auto &path = bound.get_type_path (); + visit (path); + if (path.is_marked_for_strip ()) + rust_error_at (path.get_locus (), + "cannot strip type path in this position"); +} +void +AttrVisitor::visit (AST::ImplTraitType &type) +{ + // don't strip directly, only components of bounds + for (auto &bound : type.get_type_param_bounds ()) + bound->accept_vis (*this); +} +void +AttrVisitor::visit (AST::TraitObjectType &type) +{ + // don't strip directly, only components of bounds + for (auto &bound : type.get_type_param_bounds ()) + bound->accept_vis (*this); +} +void +AttrVisitor::visit (AST::ParenthesisedType &type) +{ + // expand but don't strip inner type + auto &inner_type = type.get_type_in_parens (); + inner_type->accept_vis (*this); + if (inner_type->is_marked_for_strip ()) + rust_error_at (inner_type->get_locus (), + "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::ImplTraitTypeOneBound &type) +{ + // no stripping possible + visit (type.get_trait_bound ()); +} +void +AttrVisitor::visit (AST::TraitObjectTypeOneBound &type) +{ + // no stripping possible + visit (type.get_trait_bound ()); +} +void +AttrVisitor::visit (AST::TupleType &type) +{ + // TODO: assuming that types can't be stripped as types don't have outer + // attributes + for (auto &elem_type : type.get_elems ()) + { + elem_type->accept_vis (*this); + if (elem_type->is_marked_for_strip ()) + rust_error_at (elem_type->get_locus (), + "cannot strip type in this position"); + } +} +void +AttrVisitor::visit (AST::NeverType &) +{ + // no stripping possible +} +void +AttrVisitor::visit (AST::RawPointerType &type) +{ + // expand but don't strip type pointed to + auto &pointed_type = type.get_type_pointed_to (); + pointed_type->accept_vis (*this); + if (pointed_type->is_marked_for_strip ()) + rust_error_at (pointed_type->get_locus (), + "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::ReferenceType &type) +{ + // expand but don't strip type referenced + auto &referenced_type = type.get_type_referenced (); + referenced_type->accept_vis (*this); + if (referenced_type->is_marked_for_strip ()) + rust_error_at (referenced_type->get_locus (), + "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::ArrayType &type) +{ + // expand but don't strip type referenced + auto &base_type = type.get_elem_type (); + base_type->accept_vis (*this); + if (base_type->is_marked_for_strip ()) + rust_error_at (base_type->get_locus (), + "cannot strip type in this position"); + + // same for expression + auto &size_expr = type.get_size_expr (); + size_expr->accept_vis (*this); + if (size_expr->is_marked_for_strip ()) + rust_error_at (size_expr->get_locus (), + "cannot strip expression in this position"); +} +void +AttrVisitor::visit (AST::SliceType &type) +{ + // expand but don't strip elem type + auto &elem_type = type.get_elem_type (); + elem_type->accept_vis (*this); + if (elem_type->is_marked_for_strip ()) + rust_error_at (elem_type->get_locus (), + "cannot strip type in this position"); +} +void +AttrVisitor::visit (AST::InferredType &) +{ + // none possible +} +void +AttrVisitor::visit (AST::BareFunctionType &type) +{ + // seem to be no generics + + // presumably function params can be stripped + auto ¶ms = type.get_function_params (); + for (auto it = params.begin (); it != params.end ();) + { + auto ¶m = *it; + + auto ¶m_attrs = param.get_outer_attrs (); + expander.expand_cfg_attrs (param_attrs); + if (expander.fails_cfg_with_expand (param_attrs)) + { + it = params.erase (it); + continue; + } + + expander.push_context (MacroExpander::ContextType::TYPE); + + auto &type = param.get_type (); + type->accept_vis (*this); + + maybe_expand_type (type); + + if (type->is_marked_for_strip ()) + rust_error_at (type->get_locus (), + "cannot strip type in this position"); + + expander.pop_context (); + + // increment if nothing else happens + ++it; + } + + /* TODO: assuming that variadic nature cannot be stripped. If this + * is not true, then have code here to do so. */ + + if (type.has_return_type ()) + { + // FIXME: Can we have type expansion in this position? + // In that case, we need to handle AST::TypeNoBounds on top of just + // AST::Types + auto &return_type = type.get_return_type (); + return_type->accept_vis (*this); + if (return_type->is_marked_for_strip ()) + rust_error_at (return_type->get_locus (), + "cannot strip type in this position"); + } + + // no where clause, apparently +} +void +AttrVisitor::maybe_expand_expr (std::unique_ptr &expr) +{ + auto final_fragment = expand_macro_fragment_recursive (); + if (final_fragment.should_expand ()) + expr = final_fragment.take_expression_fragment (); +} + +void +AttrVisitor::maybe_expand_type (std::unique_ptr &type) +{ + auto final_fragment = expand_macro_fragment_recursive (); + if (final_fragment.should_expand ()) + type = final_fragment.take_type_fragment (); +} +} // namespace Rust diff --git a/gcc/rust/expand/rust-attribute-visitor.h b/gcc/rust/expand/rust-attribute-visitor.h new file mode 100644 index 00000000000..0f9d1065334 --- /dev/null +++ b/gcc/rust/expand/rust-attribute-visitor.h @@ -0,0 +1,316 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#include "rust-ast-visitor.h" +#include "rust-ast.h" +#include "rust-macro-expand.h" + +namespace Rust { +// Visitor used to expand attributes. +class AttrVisitor : public AST::ASTVisitor +{ +private: + MacroExpander &expander; + void maybe_expand_expr (std::unique_ptr &expr); + void maybe_expand_type (std::unique_ptr &expr); + +public: + AttrVisitor (MacroExpander &expander) : expander (expander) {} + + void expand_struct_fields (std::vector &fields); + void expand_tuple_fields (std::vector &fields); + void expand_function_params (std::vector ¶ms); + void expand_generic_args (AST::GenericArgs &args); + void expand_qualified_path_type (AST::QualifiedPathType &path_type); + void expand_closure_params (std::vector ¶ms); + void expand_self_param (AST::SelfParam &self_param); + void expand_where_clause (AST::WhereClause &where_clause); + void expand_trait_function_decl (AST::TraitFunctionDecl &decl); + void expand_trait_method_decl (AST::TraitMethodDecl &decl); + + /** + * Expand The current macro fragment recursively until it could not be + * expanded further. + * + * The return value checking works because correctly + * expanded fragment can never be an error (if the fragment can not be + * expanded, a stand-in error fragment will be returned; for fragments that + * could not be further expanded: the fragment prior to the expansion failure + * will be returned). + * + * @return Either the expanded fragment or an empty errored-out fragment + * indicating an expansion failure. + */ + AST::ASTFragment expand_macro_fragment_recursive () + { + auto fragment = expander.take_expanded_fragment (*this); + unsigned int original_depth = expander.expansion_depth; + auto final_fragment = AST::ASTFragment ({}, true); + + while (fragment.should_expand ()) + { + final_fragment = std::move (fragment); + expander.expansion_depth++; + // further expand the previously expanded macro fragment + auto new_fragment = expander.take_expanded_fragment (*this); + if (new_fragment.is_error ()) + break; + fragment = std::move (new_fragment); + } + expander.expansion_depth = original_depth; + return final_fragment; + } + + /** + * Expand a set of values, erasing them if they are marked for strip, and + * replacing them with expanded macro nodes if necessary. + * This function is slightly different from `expand_pointer_allow_strip` as + * it can only be called in certain expansion contexts - where macro + * invocations are allowed. + * + * @param ctx Context to use for macro expansion + * @param values Iterable reference over values to replace or erase + * @param extractor Function to call when replacing values with the content + * of an expanded AST node + */ + template + void expand_macro_children (MacroExpander::ContextType ctx, T &values, + std::function extractor) + { + expander.push_context (ctx); + + for (auto it = values.begin (); it != values.end ();) + { + auto &value = *it; + + // mark for stripping if required + value->accept_vis (*this); + + // recursively expand the children + auto final_fragment = expand_macro_fragment_recursive (); + + if (final_fragment.should_expand ()) + { + it = values.erase (it); + for (auto &node : final_fragment.get_nodes ()) + { + auto new_node = extractor (node); + if (new_node != nullptr && !new_node->is_marked_for_strip ()) + { + it = values.insert (it, std::move (new_node)); + it++; + } + } + } + else if (value->is_marked_for_strip ()) + { + it = values.erase (it); + } + else + { + ++it; + } + } + + expander.pop_context (); + } + + template void expand_pointer_allow_strip (T &values) + { + for (auto it = values.begin (); it != values.end ();) + { + auto &value = *it; + + // mark for stripping if required + value->accept_vis (*this); + if (value->is_marked_for_strip ()) + { + it = values.erase (it); + } + else + { + ++it; + } + } + } + + void visit (AST::Token &) override; + void visit (AST::DelimTokenTree &) override; + void visit (AST::AttrInputMetaItemContainer &) override; + void visit (AST::IdentifierExpr &ident_expr) override; + void visit (AST::Lifetime &) override; + void visit (AST::LifetimeParam &) override; + void visit (AST::ConstGenericParam &) override; + + void visit (AST::MacroInvocation ¯o_invoc) override; + + void visit (AST::PathInExpression &path) override; + void visit (AST::TypePathSegment &) override; + void visit (AST::TypePathSegmentGeneric &segment) override; + void visit (AST::TypePathSegmentFunction &segment) override; + void visit (AST::TypePath &path) override; + void visit (AST::QualifiedPathInExpression &path) override; + void visit (AST::QualifiedPathInType &path) override; + + void visit (AST::LiteralExpr &expr) override; + void visit (AST::AttrInputLiteral &) override; + void visit (AST::MetaItemLitExpr &) override; + void visit (AST::MetaItemPathLit &) override; + void visit (AST::BorrowExpr &expr) override; + void visit (AST::DereferenceExpr &expr) override; + void visit (AST::ErrorPropagationExpr &expr) override; + void visit (AST::NegationExpr &expr) override; + void visit (AST::ArithmeticOrLogicalExpr &expr) override; + void visit (AST::ComparisonExpr &expr) override; + void visit (AST::LazyBooleanExpr &expr) override; + void visit (AST::TypeCastExpr &expr) override; + void visit (AST::AssignmentExpr &expr) override; + void visit (AST::CompoundAssignmentExpr &expr) override; + void visit (AST::GroupedExpr &expr) override; + void visit (AST::ArrayElemsValues &elems) override; + void visit (AST::ArrayElemsCopied &elems) override; + void visit (AST::ArrayExpr &expr) override; + void visit (AST::ArrayIndexExpr &expr) override; + void visit (AST::TupleExpr &expr) override; + void visit (AST::TupleIndexExpr &expr) override; + void visit (AST::StructExprStruct &expr) override; + void visit (AST::StructExprFieldIdentifier &) override; + void visit (AST::StructExprFieldIdentifierValue &field) override; + + void visit (AST::StructExprFieldIndexValue &field) override; + void visit (AST::StructExprStructFields &expr) override; + void visit (AST::StructExprStructBase &expr) override; + void visit (AST::CallExpr &expr) override; + void visit (AST::MethodCallExpr &expr) override; + void visit (AST::FieldAccessExpr &expr) override; + void visit (AST::ClosureExprInner &expr) override; + + void visit (AST::BlockExpr &expr) override; + + void visit (AST::ClosureExprInnerTyped &expr) override; + void visit (AST::ContinueExpr &expr) override; + void visit (AST::BreakExpr &expr) override; + void visit (AST::RangeFromToExpr &expr) override; + void visit (AST::RangeFromExpr &expr) override; + void visit (AST::RangeToExpr &expr) override; + void visit (AST::RangeFullExpr &) override; + void visit (AST::RangeFromToInclExpr &expr) override; + void visit (AST::RangeToInclExpr &expr) override; + void visit (AST::ReturnExpr &expr) override; + void visit (AST::UnsafeBlockExpr &expr) override; + void visit (AST::LoopExpr &expr) override; + void visit (AST::WhileLoopExpr &expr) override; + void visit (AST::WhileLetLoopExpr &expr) override; + void visit (AST::ForLoopExpr &expr) override; + void visit (AST::IfExpr &expr) override; + void visit (AST::IfExprConseqElse &expr) override; + void visit (AST::IfExprConseqIf &expr) override; + void visit (AST::IfExprConseqIfLet &expr) override; + void visit (AST::IfLetExpr &expr) override; + void visit (AST::IfLetExprConseqElse &expr) override; + void visit (AST::IfLetExprConseqIf &expr) override; + void visit (AST::IfLetExprConseqIfLet &expr) override; + void visit (AST::MatchExpr &expr) override; + void visit (AST::AwaitExpr &expr) override; + void visit (AST::AsyncBlockExpr &expr) override; + void visit (AST::TypeParam ¶m) override; + void visit (AST::LifetimeWhereClauseItem &) override; + void visit (AST::TypeBoundWhereClauseItem &item) override; + void visit (AST::Method &method) override; + void visit (AST::Module &module) override; + void visit (AST::ExternCrate &crate) override; + void visit (AST::UseTreeGlob &) override; + void visit (AST::UseTreeList &) override; + void visit (AST::UseTreeRebind &) override; + void visit (AST::UseDeclaration &use_decl) override; + void visit (AST::Function &function) override; + void visit (AST::TypeAlias &type_alias) override; + void visit (AST::StructStruct &struct_item) override; + void visit (AST::TupleStruct &tuple_struct) override; + void visit (AST::EnumItem &item) override; + void visit (AST::EnumItemTuple &item) override; + void visit (AST::EnumItemStruct &item) override; + void visit (AST::EnumItemDiscriminant &item) override; + void visit (AST::Enum &enum_item) override; + void visit (AST::Union &union_item) override; + void visit (AST::ConstantItem &const_item) override; + void visit (AST::StaticItem &static_item) override; + void visit (AST::TraitItemFunc &item) override; + void visit (AST::TraitItemMethod &item) override; + void visit (AST::TraitItemConst &item) override; + void visit (AST::TraitItemType &item) override; + void visit (AST::Trait &trait) override; + void visit (AST::InherentImpl &impl) override; + void visit (AST::TraitImpl &impl) override; + void visit (AST::ExternalStaticItem &item) override; + void visit (AST::ExternalFunctionItem &item) override; + void visit (AST::ExternBlock &block) override; + + // I don't think it would be possible to strip macros without expansion + void visit (AST::MacroMatchFragment &) override; + void visit (AST::MacroMatchRepetition &) override; + void visit (AST::MacroMatcher &) override; + void visit (AST::MacroRulesDefinition &rules_def) override; + void visit (AST::MetaItemPath &) override; + void visit (AST::MetaItemSeq &) override; + void visit (AST::MetaWord &) override; + void visit (AST::MetaNameValueStr &) override; + void visit (AST::MetaListPaths &) override; + void visit (AST::MetaListNameValueStr &) override; + void visit (AST::LiteralPattern &) override; + void visit (AST::IdentifierPattern &pattern) override; + void visit (AST::WildcardPattern &) override; + void visit (AST::RangePatternBoundLiteral &) override; + void visit (AST::RangePatternBoundPath &bound) override; + void visit (AST::RangePatternBoundQualPath &bound) override; + void visit (AST::RangePattern &pattern) override; + void visit (AST::ReferencePattern &pattern) override; + void visit (AST::StructPatternFieldTuplePat &field) override; + void visit (AST::StructPatternFieldIdentPat &field) override; + void visit (AST::StructPatternFieldIdent &field) override; + void visit (AST::StructPattern &pattern) override; + void visit (AST::TupleStructItemsNoRange &tuple_items) override; + void visit (AST::TupleStructItemsRange &tuple_items) override; + void visit (AST::TupleStructPattern &pattern) override; + void visit (AST::TuplePatternItemsMultiple &tuple_items) override; + void visit (AST::TuplePatternItemsRanged &tuple_items) override; + void visit (AST::TuplePattern &pattern) override; + void visit (AST::GroupedPattern &pattern) override; + void visit (AST::SlicePattern &pattern) override; + + void visit (AST::EmptyStmt &) override; + void visit (AST::LetStmt &stmt) override; + void visit (AST::ExprStmtWithoutBlock &stmt) override; + void visit (AST::ExprStmtWithBlock &stmt) override; + + void visit (AST::TraitBound &bound) override; + void visit (AST::ImplTraitType &type) override; + void visit (AST::TraitObjectType &type) override; + void visit (AST::ParenthesisedType &type) override; + void visit (AST::ImplTraitTypeOneBound &type) override; + void visit (AST::TraitObjectTypeOneBound &type) override; + void visit (AST::TupleType &type) override; + void visit (AST::NeverType &) override; + void visit (AST::RawPointerType &type) override; + void visit (AST::ReferenceType &type) override; + void visit (AST::ArrayType &type) override; + void visit (AST::SliceType &type) override; + void visit (AST::InferredType &) override; + void visit (AST::BareFunctionType &type) override; +}; +} // namespace Rust diff --git a/gcc/rust/expand/rust-macro-builtins.cc b/gcc/rust/expand/rust-macro-builtins.cc new file mode 100644 index 00000000000..5eace13d197 --- /dev/null +++ b/gcc/rust/expand/rust-macro-builtins.cc @@ -0,0 +1,484 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#include "rust-macro-builtins.h" +#include "rust-diagnostics.h" +#include "rust-expr.h" +#include "rust-session-manager.h" +#include "rust-macro-invoc-lexer.h" +#include "rust-lex.h" +#include "rust-parse.h" + +namespace Rust { +namespace { +std::unique_ptr +make_string (Location locus, std::string value) +{ + return std::unique_ptr ( + new AST::LiteralExpr (value, AST::Literal::STRING, + PrimitiveCoreType::CORETYPE_STR, {}, locus)); +} + +/* Match the end token of a macro given the start delimiter of the macro */ + +static inline TokenId +macro_end_token (AST::DelimTokenTree &invoc_token_tree, + Parser &parser) +{ + auto last_token_id = TokenId::RIGHT_CURLY; + switch (invoc_token_tree.get_delim_type ()) + { + case AST::DelimType::PARENS: + last_token_id = TokenId::RIGHT_PAREN; + rust_assert (parser.skip_token (LEFT_PAREN)); + break; + + case AST::DelimType::CURLY: + rust_assert (parser.skip_token (LEFT_CURLY)); + break; + + case AST::DelimType::SQUARE: + last_token_id = TokenId::RIGHT_SQUARE; + rust_assert (parser.skip_token (LEFT_SQUARE)); + break; + } + + return last_token_id; +} + +/* Parse a single string literal from the given delimited token tree, + and return the LiteralExpr for it. Allow for an optional trailing comma, + but otherwise enforce that these are the only tokens. */ + +std::unique_ptr +parse_single_string_literal (AST::DelimTokenTree &invoc_token_tree, + Location invoc_locus) +{ + MacroInvocLexer lex (invoc_token_tree.to_token_stream ()); + Parser parser (lex); + + auto last_token_id = macro_end_token (invoc_token_tree, parser); + + std::unique_ptr lit_expr = nullptr; + + if (parser.peek_current_token ()->get_id () == STRING_LITERAL) + { + lit_expr = parser.parse_literal_expr (); + parser.maybe_skip_token (COMMA); + if (parser.peek_current_token ()->get_id () != last_token_id) + { + lit_expr = nullptr; + rust_error_at (invoc_locus, "macro takes 1 argument"); + } + } + else if (parser.peek_current_token ()->get_id () == last_token_id) + rust_error_at (invoc_locus, "macro takes 1 argument"); + else + rust_error_at (invoc_locus, "argument must be a string literal"); + + parser.skip_token (last_token_id); + + return lit_expr; +} + +/* Treat PATH as a path relative to the source file currently being + compiled, and return the absolute path for it. */ + +std::string +source_relative_path (std::string path, Location locus) +{ + std::string compile_fname + = Session::get_instance ().linemap->location_file (locus); + + auto dir_separator_pos = compile_fname.rfind (file_separator); + + /* If there is no file_separator in the path, use current dir ('.'). */ + std::string dirname; + if (dir_separator_pos == std::string::npos) + dirname = std::string (".") + file_separator; + else + dirname = compile_fname.substr (0, dir_separator_pos) + file_separator; + + return dirname + path; +} + +/* Read the full contents of the file FILENAME and return them in a vector. + FIXME: platform specific. */ + +std::vector +load_file_bytes (const char *filename) +{ + RAIIFile file_wrap (filename); + if (file_wrap.get_raw () == nullptr) + { + rust_error_at (Location (), "cannot open filename %s: %m", filename); + return std::vector (); + } + + FILE *f = file_wrap.get_raw (); + fseek (f, 0L, SEEK_END); + long fsize = ftell (f); + fseek (f, 0L, SEEK_SET); + + std::vector buf (fsize); + + if (fread (&buf[0], fsize, 1, f) != 1) + { + rust_error_at (Location (), "error reading file %s: %m", filename); + return std::vector (); + } + + return buf; +} +} // namespace + +AST::ASTFragment +MacroBuiltin::assert (Location invoc_locus, AST::MacroInvocData &invoc) +{ + rust_debug ("assert!() called"); + + return AST::ASTFragment::create_error (); +} + +AST::ASTFragment +MacroBuiltin::file (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto current_file + = Session::get_instance ().linemap->location_file (invoc_locus); + auto file_str = AST::SingleASTNode (make_string (invoc_locus, current_file)); + + return AST::ASTFragment ({file_str}); +} + +AST::ASTFragment +MacroBuiltin::column (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto current_column + = Session::get_instance ().linemap->location_to_column (invoc_locus); + + auto column_no = AST::SingleASTNode (std::unique_ptr ( + new AST::LiteralExpr (std::to_string (current_column), AST::Literal::INT, + PrimitiveCoreType::CORETYPE_U32, {}, invoc_locus))); + + return AST::ASTFragment ({column_no}); +} + +/* Expand builtin macro include_bytes!("filename"), which includes the contents + of the given file as reference to a byte array. Yields an expression of type + &'static [u8; N]. */ + +AST::ASTFragment +MacroBuiltin::include_bytes (Location invoc_locus, AST::MacroInvocData &invoc) +{ + /* Get target filename from the macro invocation, which is treated as a path + relative to the include!-ing file (currently being compiled). */ + auto lit_expr + = parse_single_string_literal (invoc.get_delim_tok_tree (), invoc_locus); + if (lit_expr == nullptr) + return AST::ASTFragment::create_error (); + + std::string target_filename + = source_relative_path (lit_expr->as_string (), invoc_locus); + + std::vector bytes = load_file_bytes (target_filename.c_str ()); + + /* Is there a more efficient way to do this? */ + std::vector> elts; + for (uint8_t b : bytes) + { + elts.emplace_back ( + new AST::LiteralExpr (std::string (1, (char) b), AST::Literal::BYTE, + PrimitiveCoreType::CORETYPE_U8, + {} /* outer_attrs */, invoc_locus)); + } + + auto elems = std::unique_ptr ( + new AST::ArrayElemsValues (std::move (elts), invoc_locus)); + + auto array = std::unique_ptr ( + new AST::ArrayExpr (std::move (elems), {}, {}, invoc_locus)); + + auto borrow = std::unique_ptr ( + new AST::BorrowExpr (std::move (array), false, false, {}, invoc_locus)); + + auto node = AST::SingleASTNode (std::move (borrow)); + return AST::ASTFragment ({node}); +} + +/* Expand builtin macro include_str!("filename"), which includes the contents + of the given file as a string. The file must be UTF-8 encoded. Yields an + expression of type &'static str. */ + +AST::ASTFragment +MacroBuiltin::include_str (Location invoc_locus, AST::MacroInvocData &invoc) +{ + /* Get target filename from the macro invocation, which is treated as a path + relative to the include!-ing file (currently being compiled). */ + auto lit_expr + = parse_single_string_literal (invoc.get_delim_tok_tree (), invoc_locus); + if (lit_expr == nullptr) + return AST::ASTFragment::create_error (); + + std::string target_filename + = source_relative_path (lit_expr->as_string (), invoc_locus); + + std::vector bytes = load_file_bytes (target_filename.c_str ()); + + /* FIXME: Enforce that the file contents are valid UTF-8. */ + std::string str ((const char *) &bytes[0], bytes.size ()); + + auto node = AST::SingleASTNode (make_string (invoc_locus, str)); + return AST::ASTFragment ({node}); +} + +/* Expand builtin macro compile_error!("error"), which forces a compile error + during the compile time. */ +AST::ASTFragment +MacroBuiltin::compile_error (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto lit_expr + = parse_single_string_literal (invoc.get_delim_tok_tree (), invoc_locus); + if (lit_expr == nullptr) + return AST::ASTFragment::create_error (); + + std::string error_string = lit_expr->as_string (); + rust_error_at (invoc_locus, "%s", error_string.c_str ()); + + return AST::ASTFragment::create_error (); +} + +/* Expand builtin macro concat!(), which joins all the literal parameters + into a string with no delimiter. */ + +AST::ASTFragment +MacroBuiltin::concat (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto invoc_token_tree = invoc.get_delim_tok_tree (); + MacroInvocLexer lex (invoc_token_tree.to_token_stream ()); + Parser parser (lex); + + auto str = std::string (); + bool has_error = false; + + auto last_token_id = macro_end_token (invoc_token_tree, parser); + + /* NOTE: concat! could accept no argument, so we don't have any checks here */ + while (parser.peek_current_token ()->get_id () != last_token_id) + { + auto lit_expr = parser.parse_literal_expr (); + if (lit_expr) + { + str += lit_expr->as_string (); + } + else + { + auto current_token = parser.peek_current_token (); + rust_error_at (current_token->get_locus (), + "argument must be a constant literal"); + has_error = true; + // Just crash if the current token can't be skipped + rust_assert (parser.skip_token (current_token->get_id ())); + } + parser.maybe_skip_token (COMMA); + } + + parser.skip_token (last_token_id); + + if (has_error) + return AST::ASTFragment::create_error (); + + auto node = AST::SingleASTNode (make_string (invoc_locus, str)); + return AST::ASTFragment ({node}); +} + +/* Expand builtin macro env!(), which inspects an environment variable at + compile time. */ + +AST::ASTFragment +MacroBuiltin::env (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto invoc_token_tree = invoc.get_delim_tok_tree (); + MacroInvocLexer lex (invoc_token_tree.to_token_stream ()); + Parser parser (lex); + + auto last_token_id = macro_end_token (invoc_token_tree, parser); + + if (parser.peek_current_token ()->get_id () != STRING_LITERAL) + { + if (parser.peek_current_token ()->get_id () == last_token_id) + rust_error_at (invoc_locus, "env! takes 1 or 2 arguments"); + else + rust_error_at (parser.peek_current_token ()->get_locus (), + "argument must be a string literal"); + return AST::ASTFragment::create_error (); + } + + auto lit_expr = parser.parse_literal_expr (); + auto comma_skipped = parser.maybe_skip_token (COMMA); + + std::unique_ptr error_expr = nullptr; + + if (parser.peek_current_token ()->get_id () != last_token_id) + { + if (!comma_skipped) + { + rust_error_at (parser.peek_current_token ()->get_locus (), + "expected token: %<,%>"); + return AST::ASTFragment::create_error (); + } + if (parser.peek_current_token ()->get_id () != STRING_LITERAL) + { + rust_error_at (parser.peek_current_token ()->get_locus (), + "argument must be a string literal"); + return AST::ASTFragment::create_error (); + } + + error_expr = parser.parse_literal_expr (); + parser.maybe_skip_token (COMMA); + } + + if (parser.peek_current_token ()->get_id () != last_token_id) + { + rust_error_at (invoc_locus, "env! takes 1 or 2 arguments"); + return AST::ASTFragment::create_error (); + } + + parser.skip_token (last_token_id); + + auto env_value = getenv (lit_expr->as_string ().c_str ()); + + if (env_value == nullptr) + { + if (error_expr == nullptr) + rust_error_at (invoc_locus, "environment variable %qs not defined", + lit_expr->as_string ().c_str ()); + else + rust_error_at (invoc_locus, "%s", error_expr->as_string ().c_str ()); + return AST::ASTFragment::create_error (); + } + + auto node = AST::SingleASTNode (make_string (invoc_locus, env_value)); + return AST::ASTFragment ({node}); +} + +AST::ASTFragment +MacroBuiltin::cfg (Location invoc_locus, AST::MacroInvocData &invoc) +{ + // only parse if not already parsed + if (!invoc.is_parsed ()) + { + std::unique_ptr converted_input ( + invoc.get_delim_tok_tree ().parse_to_meta_item ()); + + if (converted_input == nullptr) + { + rust_debug ("DEBUG: failed to parse macro to meta item"); + // TODO: do something now? is this an actual error? + } + else + { + std::vector> meta_items ( + std::move (converted_input->get_items ())); + invoc.set_meta_item_output (std::move (meta_items)); + } + } + + /* TODO: assuming that cfg! macros can only have one meta item inner, like cfg + * attributes */ + if (invoc.get_meta_items ().size () != 1) + return AST::ASTFragment::create_error (); + + bool result = invoc.get_meta_items ()[0]->check_cfg_predicate ( + Session::get_instance ()); + auto literal_exp = AST::SingleASTNode (std::unique_ptr ( + new AST::LiteralExpr (result ? "true" : "false", AST::Literal::BOOL, + PrimitiveCoreType::CORETYPE_BOOL, {}, invoc_locus))); + + return AST::ASTFragment ({literal_exp}); +} + +/* Expand builtin macro include!(), which includes a source file at the current + scope compile time. */ + +AST::ASTFragment +MacroBuiltin::include (Location invoc_locus, AST::MacroInvocData &invoc) +{ + /* Get target filename from the macro invocation, which is treated as a path + relative to the include!-ing file (currently being compiled). */ + auto lit_expr + = parse_single_string_literal (invoc.get_delim_tok_tree (), invoc_locus); + if (lit_expr == nullptr) + return AST::ASTFragment::create_error (); + + std::string filename + = source_relative_path (lit_expr->as_string (), invoc_locus); + auto target_filename + = Rust::Session::get_instance ().include_extra_file (std::move (filename)); + + RAIIFile target_file (target_filename); + Linemap *linemap = Session::get_instance ().linemap; + + if (!target_file.ok ()) + { + rust_error_at (lit_expr->get_locus (), + "cannot open included file %qs: %m", target_filename); + return AST::ASTFragment::create_error (); + } + + rust_debug ("Attempting to parse included file %s", target_filename); + + Lexer lex (target_filename, std::move (target_file), linemap); + Parser parser (lex); + + auto parsed_items = parser.parse_items (); + bool has_error = !parser.get_errors ().empty (); + + for (const auto &error : parser.get_errors ()) + error.emit_error (); + + if (has_error) + { + // inform the user that the errors above are from a included file + rust_inform (invoc_locus, "included from here"); + return AST::ASTFragment::create_error (); + } + + std::vector nodes{}; + for (auto &item : parsed_items) + { + AST::SingleASTNode node (std::move (item)); + nodes.push_back (node); + } + + return AST::ASTFragment (nodes); +} + +AST::ASTFragment +MacroBuiltin::line (Location invoc_locus, AST::MacroInvocData &invoc) +{ + auto current_line + = Session::get_instance ().linemap->location_to_line (invoc_locus); + + auto line_no = AST::SingleASTNode (std::unique_ptr ( + new AST::LiteralExpr (std::to_string (current_line), AST::Literal::INT, + PrimitiveCoreType::CORETYPE_U32, {}, invoc_locus))); + + return AST::ASTFragment ({line_no}); +} + +} // namespace Rust diff --git a/gcc/rust/expand/rust-macro-builtins.h b/gcc/rust/expand/rust-macro-builtins.h new file mode 100644 index 00000000000..91f3727d450 --- /dev/null +++ b/gcc/rust/expand/rust-macro-builtins.h @@ -0,0 +1,107 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#ifndef RUST_MACRO_BUILTINS_H +#define RUST_MACRO_BUILTINS_H + +#include "rust-ast.h" +#include "rust-location.h" + +/** + * This class provides a list of builtin macros implemented by the compiler. + * The functions defined are called "builtin transcribers" in that they replace + * the transcribing part of a macro definition. + * + * Like regular macro transcribers, they are responsible for building and + * returning an AST fragment: basically a vector of AST nodes put together. + * + * Unlike regular declarative macros where each match arm has its own associated + * transcriber, builtin transcribers are responsible for handling all match arms + * of the macro. This means that you should take extra care when implementing a + * builtin containing multiple match arms: You will probably need to do some + * lookahead in order to determine which match arm the user intended to use. + * + * An example of this is the `assert!()` macro: + * + * ``` + * macro_rules! assert { + * ($cond:expr $(,)?) => {{ ... }}; + * ($cond : expr, $ ($arg : tt) +) = > {{ ... }}; + * } + * ``` + * + * If more tokens exist beyond the optional comma, they need to be handled as + * a token-tree for a custom panic message. + * + * These builtin macros with empty transcribers are defined in the standard + * library. They are marked with a special attribute, `#[rustc_builtin_macro]`. + * When this attribute is present on a macro definition, the compiler should + * look for an associated transcriber in the mappings. Meaning that you must + * remember to insert your transcriber in the `builtin_macros` map of the + *`Mappings`. + * + * This map is built as a static variable in the `insert_macro_def()` method + * of the `Mappings` class. + */ + +/* If assert is defined as a macro this file will not parse, so undefine this + before continuing. */ +#ifdef assert +#undef assert +#endif + +namespace Rust { +class MacroBuiltin +{ +public: + static AST::ASTFragment assert (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment file (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment column (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment include_bytes (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment include_str (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment compile_error (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment concat (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment env (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment cfg (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment include (Location invoc_locus, + AST::MacroInvocData &invoc); + + static AST::ASTFragment line (Location invoc_locus, + AST::MacroInvocData &invoc); +}; +} // namespace Rust + +#endif // RUST_MACRO_BUILTINS_H diff --git a/gcc/rust/expand/rust-macro-expand.cc b/gcc/rust/expand/rust-macro-expand.cc new file mode 100644 index 00000000000..1d57e394220 --- /dev/null +++ b/gcc/rust/expand/rust-macro-expand.cc @@ -0,0 +1,1012 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#include "rust-macro-expand.h" +#include "rust-macro-substitute-ctx.h" +#include "rust-ast-full.h" +#include "rust-ast-visitor.h" +#include "rust-diagnostics.h" +#include "rust-parse.h" +#include "rust-attribute-visitor.h" + +namespace Rust { +AST::ASTFragment +MacroExpander::expand_decl_macro (Location invoc_locus, + AST::MacroInvocData &invoc, + AST::MacroRulesDefinition &rules_def, + bool semicolon) +{ + // ensure that both invocation and rules are in a valid state + rust_assert (!invoc.is_marked_for_strip ()); + rust_assert (!rules_def.is_marked_for_strip ()); + rust_assert (rules_def.get_macro_rules ().size () > 0); + + /* probably something here about parsing invoc and rules def token trees to + * token stream. if not, how would parser handle the captures of exprs and + * stuff? on the other hand, token trees may be kind of useful in rules def as + * creating a point where recursion can occur (like having + * "compare_macro_match" and then it calling itself when it finds delimiters) + */ + + /* find matching rule to invoc token tree, based on macro rule's matcher. if + * none exist, error. + * - specifically, check each matcher in order. if one fails to match, move + * onto next. */ + /* TODO: does doing this require parsing expressions and whatever in the + * invoc? if so, might as well save the results if referenced using $ or + * whatever. If not, do another pass saving them. Except this is probably + * useless as different rules could have different starting points for exprs + * or whatever. Decision trees could avoid this, but they have their own + * issues. */ + /* TODO: will need to modify the parser so that it can essentially "catch" + * errors - maybe "try_parse_expr" or whatever methods. */ + // this technically creates a back-tracking parser - this will be the + // implementation style + + /* then, after results are saved, generate the macro output from the + * transcriber token tree. if i understand this correctly, the macro + * invocation gets replaced by the transcriber tokens, except with + * substitutions made (e.g. for $i variables) */ + + /* TODO: it is probably better to modify AST::Token to store a pointer to a + * Lexer::Token (rather than being converted) - i.e. not so much have + * AST::Token as a Token but rather a TokenContainer (as it is another type of + * TokenTree). This will prevent re-conversion of Tokens between each type + * all the time, while still allowing the heterogenous storage of token trees. + */ + + AST::DelimTokenTree &invoc_token_tree = invoc.get_delim_tok_tree (); + + // find matching arm + AST::MacroRule *matched_rule = nullptr; + std::map matched_fragments; + for (auto &rule : rules_def.get_rules ()) + { + sub_stack.push (); + bool did_match_rule = try_match_rule (rule, invoc_token_tree); + matched_fragments = sub_stack.pop (); + + if (did_match_rule) + { + // // Debugging + // for (auto &kv : matched_fragments) + // rust_debug ("[fragment]: %s (%ld - %s)", kv.first.c_str (), + // kv.second.get_fragments ().size (), + // kv.second.get_kind () + // == MatchedFragmentContainer::Kind::Repetition + // ? "repetition" + // : "metavar"); + + matched_rule = &rule; + break; + } + } + + if (matched_rule == nullptr) + { + RichLocation r (invoc_locus); + r.add_range (rules_def.get_locus ()); + rust_error_at (r, "Failed to match any rule within macro"); + return AST::ASTFragment::create_error (); + } + + return transcribe_rule (*matched_rule, invoc_token_tree, matched_fragments, + semicolon, peek_context ()); +} + +void +MacroExpander::expand_invoc (AST::MacroInvocation &invoc, bool has_semicolon) +{ + if (depth_exceeds_recursion_limit ()) + { + rust_error_at (invoc.get_locus (), "reached recursion limit"); + return; + } + + AST::MacroInvocData &invoc_data = invoc.get_invoc_data (); + + // ?? + // switch on type of macro: + // - '!' syntax macro (inner switch) + // - procedural macro - "A token-based function-like macro" + // - 'macro_rules' (by example/pattern-match) macro? or not? "an + // AST-based function-like macro" + // - else is unreachable + // - attribute syntax macro (inner switch) + // - procedural macro attribute syntax - "A token-based attribute + // macro" + // - legacy macro attribute syntax? - "an AST-based attribute macro" + // - non-macro attribute: mark known + // - else is unreachable + // - derive macro (inner switch) + // - derive or legacy derive - "token-based" vs "AST-based" + // - else is unreachable + // - derive container macro - unreachable + + // lookup the rules for this macro + NodeId resolved_node = UNKNOWN_NODEID; + NodeId source_node = UNKNOWN_NODEID; + if (has_semicolon) + source_node = invoc.get_macro_node_id (); + else + source_node = invoc.get_pattern_node_id (); + auto seg + = Resolver::CanonicalPath::new_seg (source_node, + invoc_data.get_path ().as_string ()); + + bool found = resolver->get_macro_scope ().lookup (seg, &resolved_node); + if (!found) + { + rust_error_at (invoc.get_locus (), "unknown macro: [%s]", + seg.get ().c_str ()); + return; + } + + // lookup the rules + AST::MacroRulesDefinition *rules_def = nullptr; + bool ok = mappings->lookup_macro_def (resolved_node, &rules_def); + rust_assert (ok); + + auto fragment = AST::ASTFragment::create_error (); + + if (rules_def->is_builtin ()) + fragment + = rules_def->get_builtin_transcriber () (invoc.get_locus (), invoc_data); + else + fragment = expand_decl_macro (invoc.get_locus (), invoc_data, *rules_def, + has_semicolon); + + set_expanded_fragment (std::move (fragment)); +} + +/* Determines whether any cfg predicate is false and hence item with attributes + * should be stripped. Note that attributes must be expanded before calling. */ +bool +MacroExpander::fails_cfg (const AST::AttrVec &attrs) const +{ + for (const auto &attr : attrs) + { + if (attr.get_path () == "cfg" && !attr.check_cfg_predicate (session)) + return true; + } + return false; +} + +/* Determines whether any cfg predicate is false and hence item with attributes + * should be stripped. Will expand attributes as well. */ +bool +MacroExpander::fails_cfg_with_expand (AST::AttrVec &attrs) const +{ + // TODO: maybe have something that strips cfg attributes that evaluate true? + for (auto &attr : attrs) + { + if (attr.get_path () == "cfg") + { + if (!attr.is_parsed_to_meta_item ()) + attr.parse_attr_to_meta_item (); + + // DEBUG + if (!attr.is_parsed_to_meta_item ()) + rust_debug ("failed to parse attr to meta item, right before " + "cfg predicate check"); + else + rust_debug ("attr has been successfully parsed to meta item, " + "right before cfg predicate check"); + + if (!attr.check_cfg_predicate (session)) + { + // DEBUG + rust_debug ( + "cfg predicate failed for attribute: \033[0;31m'%s'\033[0m", + attr.as_string ().c_str ()); + + return true; + } + else + { + // DEBUG + rust_debug ("cfg predicate succeeded for attribute: " + "\033[0;31m'%s'\033[0m", + attr.as_string ().c_str ()); + } + } + } + return false; +} + +// Expands cfg_attr attributes. +void +MacroExpander::expand_cfg_attrs (AST::AttrVec &attrs) +{ + for (std::size_t i = 0; i < attrs.size (); i++) + { + auto &attr = attrs[i]; + if (attr.get_path () == "cfg_attr") + { + if (!attr.is_parsed_to_meta_item ()) + attr.parse_attr_to_meta_item (); + + if (attr.check_cfg_predicate (session)) + { + // split off cfg_attr + AST::AttrVec new_attrs = attr.separate_cfg_attrs (); + + // remove attr from vector + attrs.erase (attrs.begin () + i); + + // add new attrs to vector + attrs.insert (attrs.begin () + i, + std::make_move_iterator (new_attrs.begin ()), + std::make_move_iterator (new_attrs.end ())); + } + + /* do something - if feature (first token in tree) is in fact enabled, + * make tokens listed afterwards into attributes. i.e.: for + * [cfg_attr(feature = "wow", wow1, wow2)], if "wow" is true, then add + * attributes [wow1] and [wow2] to attribute list. This can also be + * recursive, so check for expanded attributes being recursive and + * possibly recursively call the expand_attrs? */ + } + else + { + i++; + } + } + attrs.shrink_to_fit (); +} + +void +MacroExpander::expand_crate () +{ + NodeId scope_node_id = crate.get_node_id (); + resolver->get_macro_scope ().push (scope_node_id); + + /* fill macro/decorator map from init list? not sure where init list comes + * from? */ + + // TODO: does cfg apply for inner attributes? research. + // the apparent answer (from playground test) is yes + + // expand crate cfg_attr attributes + expand_cfg_attrs (crate.inner_attrs); + + if (fails_cfg_with_expand (crate.inner_attrs)) + { + // basically, delete whole crate + crate.strip_crate (); + // TODO: maybe create warning here? probably not desired behaviour + } + // expand module attributes? + + push_context (ITEM); + + // expand attributes recursively and strip items if required + AttrVisitor attr_visitor (*this); + auto &items = crate.items; + for (auto it = items.begin (); it != items.end ();) + { + auto &item = *it; + + // mark for stripping if required + item->accept_vis (attr_visitor); + + auto fragment = take_expanded_fragment (attr_visitor); + if (fragment.should_expand ()) + { + // Remove the current expanded invocation + it = items.erase (it); + for (auto &node : fragment.get_nodes ()) + { + it = items.insert (it, node.take_item ()); + it++; + } + } + else if (item->is_marked_for_strip ()) + it = items.erase (it); + else + it++; + } + + pop_context (); + + // TODO: should recursive attribute and macro expansion be done in the same + // transversal? Or in separate ones like currently? + + // expand module tree recursively + + // post-process + + // extract exported macros? +} + +bool +MacroExpander::depth_exceeds_recursion_limit () const +{ + return expansion_depth >= cfg.recursion_limit; +} + +bool +MacroExpander::try_match_rule (AST::MacroRule &match_rule, + AST::DelimTokenTree &invoc_token_tree) +{ + MacroInvocLexer lex (invoc_token_tree.to_token_stream ()); + Parser parser (lex); + + AST::MacroMatcher &matcher = match_rule.get_matcher (); + + expansion_depth++; + if (!match_matcher (parser, matcher)) + { + expansion_depth--; + return false; + } + expansion_depth--; + + bool used_all_input_tokens = parser.skip_token (END_OF_FILE); + return used_all_input_tokens; +} + +bool +MacroExpander::match_fragment (Parser &parser, + AST::MacroMatchFragment &fragment) +{ + switch (fragment.get_frag_spec ().get_kind ()) + { + case AST::MacroFragSpec::EXPR: + parser.parse_expr (); + break; + + case AST::MacroFragSpec::BLOCK: + parser.parse_block_expr (); + break; + + case AST::MacroFragSpec::IDENT: + parser.parse_identifier_pattern (); + break; + + case AST::MacroFragSpec::LITERAL: + parser.parse_literal_expr (); + break; + + case AST::MacroFragSpec::ITEM: + parser.parse_item (false); + break; + + case AST::MacroFragSpec::TY: + parser.parse_type (); + break; + + case AST::MacroFragSpec::PAT: + parser.parse_pattern (); + break; + + case AST::MacroFragSpec::PATH: + parser.parse_path_in_expression (); + break; + + case AST::MacroFragSpec::VIS: + parser.parse_visibility (); + break; + + case AST::MacroFragSpec::STMT: { + auto restrictions = ParseRestrictions (); + restrictions.consume_semi = false; + parser.parse_stmt (restrictions); + break; + } + + case AST::MacroFragSpec::LIFETIME: + parser.parse_lifetime_params (); + break; + + // is meta attributes? + case AST::MacroFragSpec::META: + parser.parse_attribute_body (); + break; + + case AST::MacroFragSpec::TT: + parser.parse_token_tree (); + break; + + // i guess we just ignore invalid and just error out + case AST::MacroFragSpec::INVALID: + return false; + } + + // it matches if the parser did not produce errors trying to parse that type + // of item + return !parser.has_errors (); +} + +bool +MacroExpander::match_matcher (Parser &parser, + AST::MacroMatcher &matcher) +{ + if (depth_exceeds_recursion_limit ()) + { + rust_error_at (matcher.get_match_locus (), "reached recursion limit"); + return false; + } + + auto delimiter = parser.peek_current_token (); + + // this is used so we can check that we delimit the stream correctly. + switch (delimiter->get_id ()) + { + case LEFT_PAREN: { + if (!parser.skip_token (LEFT_PAREN)) + return false; + } + break; + + case LEFT_SQUARE: { + if (!parser.skip_token (LEFT_SQUARE)) + return false; + } + break; + + case LEFT_CURLY: { + if (!parser.skip_token (LEFT_CURLY)) + return false; + } + break; + default: + gcc_unreachable (); + } + + const MacroInvocLexer &source = parser.get_token_source (); + + for (auto &match : matcher.get_matches ()) + { + size_t offs_begin = source.get_offs (); + + switch (match->get_macro_match_type ()) + { + case AST::MacroMatch::MacroMatchType::Fragment: { + AST::MacroMatchFragment *fragment + = static_cast (match.get ()); + if (!match_fragment (parser, *fragment)) + return false; + + // matched fragment get the offset in the token stream + size_t offs_end = source.get_offs (); + sub_stack.insert_metavar ( + MatchedFragment (fragment->get_ident (), offs_begin, offs_end)); + } + break; + + case AST::MacroMatch::MacroMatchType::Tok: { + AST::Token *tok = static_cast (match.get ()); + if (!match_token (parser, *tok)) + return false; + } + break; + + case AST::MacroMatch::MacroMatchType::Repetition: { + AST::MacroMatchRepetition *rep + = static_cast (match.get ()); + if (!match_repetition (parser, *rep)) + return false; + } + break; + + case AST::MacroMatch::MacroMatchType::Matcher: { + AST::MacroMatcher *m + = static_cast (match.get ()); + expansion_depth++; + if (!match_matcher (parser, *m)) + { + expansion_depth--; + return false; + } + expansion_depth--; + } + break; + } + } + + switch (delimiter->get_id ()) + { + case LEFT_PAREN: { + if (!parser.skip_token (RIGHT_PAREN)) + return false; + } + break; + + case LEFT_SQUARE: { + if (!parser.skip_token (RIGHT_SQUARE)) + return false; + } + break; + + case LEFT_CURLY: { + if (!parser.skip_token (RIGHT_CURLY)) + return false; + } + break; + default: + gcc_unreachable (); + } + + return true; +} + +bool +MacroExpander::match_token (Parser &parser, AST::Token &token) +{ + // FIXME this needs to actually match the content and the type + return parser.skip_token (token.get_id ()); +} + +bool +MacroExpander::match_n_matches (Parser &parser, + AST::MacroMatchRepetition &rep, + size_t &match_amount, size_t lo_bound, + size_t hi_bound) +{ + match_amount = 0; + auto &matches = rep.get_matches (); + + const MacroInvocLexer &source = parser.get_token_source (); + while (true) + { + // If the current token is a closing macro delimiter, break away. + // TODO: Is this correct? + auto t_id = parser.peek_current_token ()->get_id (); + if (t_id == RIGHT_PAREN || t_id == RIGHT_SQUARE || t_id == RIGHT_CURLY) + break; + + // Skip parsing a separator on the first match, otherwise consume it. + // If it isn't present, this is an error + if (rep.has_sep () && match_amount > 0) + if (!match_token (parser, *rep.get_sep ())) + break; + + bool valid_current_match = false; + for (auto &match : matches) + { + size_t offs_begin = source.get_offs (); + switch (match->get_macro_match_type ()) + { + case AST::MacroMatch::MacroMatchType::Fragment: { + AST::MacroMatchFragment *fragment + = static_cast (match.get ()); + valid_current_match = match_fragment (parser, *fragment); + + // matched fragment get the offset in the token stream + size_t offs_end = source.get_offs (); + + // The main difference with match_matcher happens here: Instead + // of inserting a new fragment, we append to one. If that + // fragment does not exist, then the operation is similar to + // `insert_fragment` with the difference that we are not + // creating a metavariable, but a repetition of one, which is + // really different. + sub_stack.append_fragment ( + MatchedFragment (fragment->get_ident (), offs_begin, + offs_end)); + } + break; + + case AST::MacroMatch::MacroMatchType::Tok: { + AST::Token *tok = static_cast (match.get ()); + valid_current_match = match_token (parser, *tok); + } + break; + + case AST::MacroMatch::MacroMatchType::Repetition: { + AST::MacroMatchRepetition *rep + = static_cast (match.get ()); + valid_current_match = match_repetition (parser, *rep); + } + break; + + case AST::MacroMatch::MacroMatchType::Matcher: { + AST::MacroMatcher *m + = static_cast (match.get ()); + valid_current_match = match_matcher (parser, *m); + } + break; + } + } + // If we've encountered an error once, stop trying to match more + // repetitions + if (!valid_current_match) + break; + + match_amount++; + + // Break early if we notice there's too many expressions already + if (hi_bound && match_amount > hi_bound) + break; + } + + // Check if the amount of matches we got is valid: Is it more than the lower + // bound and less than the higher bound? + bool did_meet_lo_bound = match_amount >= lo_bound; + bool did_meet_hi_bound = hi_bound ? match_amount <= hi_bound : true; + + // If the end-result is valid, then we can clear the parse errors: Since + // repetitions are parsed eagerly, it is okay to fail in some cases + auto res = did_meet_lo_bound && did_meet_hi_bound; + if (res) + parser.clear_errors (); + + return res; +} + +bool +MacroExpander::match_repetition (Parser &parser, + AST::MacroMatchRepetition &rep) +{ + size_t match_amount = 0; + bool res = false; + + std::string lo_str; + std::string hi_str; + switch (rep.get_op ()) + { + case AST::MacroMatchRepetition::MacroRepOp::ANY: + lo_str = "0"; + hi_str = "+inf"; + res = match_n_matches (parser, rep, match_amount); + break; + case AST::MacroMatchRepetition::MacroRepOp::ONE_OR_MORE: + lo_str = "1"; + hi_str = "+inf"; + res = match_n_matches (parser, rep, match_amount, 1); + break; + case AST::MacroMatchRepetition::MacroRepOp::ZERO_OR_ONE: + lo_str = "0"; + hi_str = "1"; + res = match_n_matches (parser, rep, match_amount, 0, 1); + break; + default: + gcc_unreachable (); + } + + if (!res) + rust_error_at (rep.get_match_locus (), + "invalid amount of matches for macro invocation. Expected " + "between %s and %s, got %lu", + lo_str.c_str (), hi_str.c_str (), + (unsigned long) match_amount); + + rust_debug_loc (rep.get_match_locus (), "%s matched %lu times", + res ? "successfully" : "unsuccessfully", + (unsigned long) match_amount); + + // We have to handle zero fragments differently: They will not have been + // "matched" but they are still valid and should be inserted as a special + // case. So we go through the stack map, and for every fragment which doesn't + // exist, insert a zero-matched fragment. + auto &stack_map = sub_stack.peek (); + for (auto &match : rep.get_matches ()) + { + if (match->get_macro_match_type () + == AST::MacroMatch::MacroMatchType::Fragment) + { + auto fragment = static_cast (match.get ()); + auto it = stack_map.find (fragment->get_ident ()); + + if (it == stack_map.end ()) + sub_stack.insert_matches (fragment->get_ident (), + MatchedFragmentContainer::zero ()); + } + } + + return res; +} + +/** + * Helper function to refactor calling a parsing function 0 or more times + */ +static AST::ASTFragment +parse_many (Parser &parser, TokenId &delimiter, + std::function parse_fn) +{ + std::vector nodes; + while (true) + { + if (parser.peek_current_token ()->get_id () == delimiter) + break; + + auto node = parse_fn (); + nodes.emplace_back (std::move (node)); + } + + return AST::ASTFragment (std::move (nodes)); +} + +/** + * Transcribe 0 or more items from a macro invocation + * + * @param parser Parser to extract items from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_items (Parser &parser, TokenId &delimiter) +{ + return parse_many (parser, delimiter, [&parser] () { + auto item = parser.parse_item (true); + return AST::SingleASTNode (std::move (item)); + }); +} + +/** + * Transcribe 0 or more external items from a macro invocation + * + * @param parser Parser to extract items from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_ext (Parser &parser, TokenId &delimiter) +{ + return parse_many (parser, delimiter, [&parser] () { + auto item = parser.parse_external_item (); + return AST::SingleASTNode (std::move (item)); + }); +} + +/** + * Transcribe 0 or more trait items from a macro invocation + * + * @param parser Parser to extract items from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_trait_items (Parser &parser, + TokenId &delimiter) +{ + return parse_many (parser, delimiter, [&parser] () { + auto item = parser.parse_trait_item (); + return AST::SingleASTNode (std::move (item)); + }); +} + +/** + * Transcribe 0 or more impl items from a macro invocation + * + * @param parser Parser to extract items from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_impl_items (Parser &parser, TokenId &delimiter) +{ + return parse_many (parser, delimiter, [&parser] () { + auto item = parser.parse_inherent_impl_item (); + return AST::SingleASTNode (std::move (item)); + }); +} + +/** + * Transcribe 0 or more trait impl items from a macro invocation + * + * @param parser Parser to extract items from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_trait_impl_items (Parser &parser, + TokenId &delimiter) +{ + return parse_many (parser, delimiter, [&parser] () { + auto item = parser.parse_trait_impl_item (); + return AST::SingleASTNode (std::move (item)); + }); +} + +/** + * Transcribe 0 or more statements from a macro invocation + * + * @param parser Parser to extract statements from + * @param delimiter Id of the token on which parsing should stop + */ +static AST::ASTFragment +transcribe_many_stmts (Parser &parser, TokenId &delimiter) +{ + auto restrictions = ParseRestrictions (); + restrictions.consume_semi = false; + + // FIXME: This is invalid! It needs to also handle cases where the macro + // transcriber is an expression, but since the macro call is followed by + // a semicolon, it's a valid ExprStmt + return parse_many (parser, delimiter, [&parser, restrictions] () { + auto stmt = parser.parse_stmt (restrictions); + return AST::SingleASTNode (std::move (stmt)); + }); +} + +/** + * Transcribe one expression from a macro invocation + * + * @param parser Parser to extract statements from + */ +static AST::ASTFragment +transcribe_expression (Parser &parser) +{ + auto expr = parser.parse_expr (); + + return AST::ASTFragment ({std::move (expr)}); +} + +/** + * Transcribe one type from a macro invocation + * + * @param parser Parser to extract statements from + */ +static AST::ASTFragment +transcribe_type (Parser &parser) +{ + auto type = parser.parse_type (); + + return AST::ASTFragment ({std::move (type)}); +} + +static AST::ASTFragment +transcribe_on_delimiter (Parser &parser, bool semicolon, + AST::DelimType delimiter, TokenId last_token_id) +{ + if (semicolon || delimiter == AST::DelimType::CURLY) + return transcribe_many_stmts (parser, last_token_id); + else + return transcribe_expression (parser); +} // namespace Rust + +static AST::ASTFragment +transcribe_context (MacroExpander::ContextType ctx, + Parser &parser, bool semicolon, + AST::DelimType delimiter, TokenId last_token_id) +{ + // The flow-chart in order to choose a parsing function is as follows: + // + // [switch special context] + // -- Item --> parser.parse_item(); + // -- Trait --> parser.parse_trait_item(); + // -- Impl --> parser.parse_impl_item(); + // -- Extern --> parser.parse_extern_item(); + // -- None --> [has semicolon?] + // -- Yes --> parser.parse_stmt(); + // -- No --> [switch invocation.delimiter()] + // -- { } --> parser.parse_stmt(); + // -- _ --> parser.parse_expr(); // once! + + // If there is a semicolon OR we are expanding a MacroInvocationSemi, then + // we can parse multiple items. Otherwise, parse *one* expression + + switch (ctx) + { + case MacroExpander::ContextType::ITEM: + return transcribe_many_items (parser, last_token_id); + break; + case MacroExpander::ContextType::TRAIT: + return transcribe_many_trait_items (parser, last_token_id); + break; + case MacroExpander::ContextType::IMPL: + return transcribe_many_impl_items (parser, last_token_id); + break; + case MacroExpander::ContextType::TRAIT_IMPL: + return transcribe_many_trait_impl_items (parser, last_token_id); + break; + case MacroExpander::ContextType::EXTERN: + return transcribe_many_ext (parser, last_token_id); + break; + case MacroExpander::ContextType::TYPE: + return transcribe_type (parser); + break; + default: + return transcribe_on_delimiter (parser, semicolon, delimiter, + last_token_id); + } +} + +static std::string +tokens_to_str (std::vector> &tokens) +{ + std::string str; + if (!tokens.empty ()) + { + str += tokens[0]->as_string (); + for (size_t i = 1; i < tokens.size (); i++) + str += " " + tokens[i]->as_string (); + } + + return str; +} + +AST::ASTFragment +MacroExpander::transcribe_rule ( + AST::MacroRule &match_rule, AST::DelimTokenTree &invoc_token_tree, + std::map &matched_fragments, + bool semicolon, ContextType ctx) +{ + // we can manipulate the token tree to substitute the dollar identifiers so + // that when we call parse its already substituted for us + AST::MacroTranscriber &transcriber = match_rule.get_transcriber (); + AST::DelimTokenTree &transcribe_tree = transcriber.get_token_tree (); + + auto invoc_stream = invoc_token_tree.to_token_stream (); + auto macro_rule_tokens = transcribe_tree.to_token_stream (); + + auto substitute_context + = SubstituteCtx (invoc_stream, macro_rule_tokens, matched_fragments); + std::vector> substituted_tokens + = substitute_context.substitute_tokens (); + + rust_debug ("substituted tokens: %s", + tokens_to_str (substituted_tokens).c_str ()); + + // parse it to an ASTFragment + MacroInvocLexer lex (std::move (substituted_tokens)); + Parser parser (lex); + + auto last_token_id = TokenId::RIGHT_CURLY; + + // this is used so we can check that we delimit the stream correctly. + switch (transcribe_tree.get_delim_type ()) + { + case AST::DelimType::PARENS: + last_token_id = TokenId::RIGHT_PAREN; + rust_assert (parser.skip_token (LEFT_PAREN)); + break; + + case AST::DelimType::CURLY: + rust_assert (parser.skip_token (LEFT_CURLY)); + break; + + case AST::DelimType::SQUARE: + last_token_id = TokenId::RIGHT_SQUARE; + rust_assert (parser.skip_token (LEFT_SQUARE)); + break; + } + + // see https://github.com/Rust-GCC/gccrs/issues/22 + // TL;DR: + // - Treat all macro invocations with parentheses, (), or square brackets, + // [], as expressions. + // - If the macro invocation has curly brackets, {}, it may be parsed as a + // statement depending on the context. + // - If the macro invocation has a semicolon at the end, it must be parsed + // as a statement (either via ExpressionStatement or + // MacroInvocationWithSemi) + + auto fragment + = transcribe_context (ctx, parser, semicolon, + invoc_token_tree.get_delim_type (), last_token_id); + + // emit any errors + if (parser.has_errors ()) + { + for (auto &err : parser.get_errors ()) + rust_error_at (err.locus, "%s", err.message.c_str ()); + return AST::ASTFragment::create_error (); + } + + // are all the tokens used? + bool did_delimit = parser.skip_token (last_token_id); + + bool reached_end_of_stream = did_delimit && parser.skip_token (END_OF_FILE); + if (!reached_end_of_stream) + { + const_TokenPtr current_token = parser.peek_current_token (); + rust_error_at (current_token->get_locus (), + "tokens here and after are unparsed"); + } + + return fragment; +} +} // namespace Rust diff --git a/gcc/rust/expand/rust-macro-expand.h b/gcc/rust/expand/rust-macro-expand.h new file mode 100644 index 00000000000..94d6702ecb8 --- /dev/null +++ b/gcc/rust/expand/rust-macro-expand.h @@ -0,0 +1,366 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#ifndef RUST_MACRO_EXPAND_H +#define RUST_MACRO_EXPAND_H + +#include "rust-buffered-queue.h" +#include "rust-parse.h" +#include "rust-token.h" +#include "rust-ast.h" +#include "rust-macro.h" +#include "rust-hir-map.h" +#include "rust-name-resolver.h" +#include "rust-macro-invoc-lexer.h" + +// Provides objects and method prototypes for macro expansion + +namespace Rust { +// forward decls for AST +namespace AST { +class MacroInvocation; +} + +// Object used to store configuration data for macro expansion. +// NOTE: Keep all these items complying with the latest rustc. +struct ExpansionCfg +{ + // features? + // TODO: Add `features' when we have it. + unsigned int recursion_limit = 1024; + bool trace_mac = false; // trace macro + bool should_test = false; // strip #[test] nodes if false + bool keep_macs = false; // keep macro definitions + std::string crate_name = ""; +}; + +struct MatchedFragment +{ + std::string fragment_ident; + size_t token_offset_begin; + size_t token_offset_end; + + MatchedFragment (std::string identifier, size_t token_offset_begin, + size_t token_offset_end) + : fragment_ident (identifier), token_offset_begin (token_offset_begin), + token_offset_end (token_offset_end) + {} + + /** + * Empty constructor for uninitialized fragments + */ + MatchedFragment () : MatchedFragment ("", 0, 0) {} + + std::string as_string () const + { + return fragment_ident + "=" + std::to_string (token_offset_begin) + ":" + + std::to_string (token_offset_end); + } +}; + +class MatchedFragmentContainer +{ +public: + // Does the container refer to a simple metavariable, different from a + // repetition repeated once + enum class Kind + { + MetaVar, + Repetition, + }; + + MatchedFragmentContainer (std::vector fragments, + Kind kind = Kind::Repetition) + : fragments (fragments), kind (kind) + {} + + /** + * Create a valid fragment matched zero times. This is useful for repetitions + * which allow the absence of a fragment, such as * and ? + */ + static MatchedFragmentContainer zero () + { + return MatchedFragmentContainer ({}); + } + + /** + * Create a valid fragment matched one time + */ + static MatchedFragmentContainer metavar (MatchedFragment fragment) + { + return MatchedFragmentContainer ({fragment}, Kind::MetaVar); + } + + /** + * Add a matched fragment to the container + */ + void add_fragment (MatchedFragment fragment) + { + rust_assert (!is_single_fragment ()); + + fragments.emplace_back (fragment); + } + + size_t get_match_amount () const { return fragments.size (); } + const std::vector &get_fragments () const + { + return fragments; + } + // const std::string &get_fragment_name () const { return fragment_name; } + + bool is_single_fragment () const + { + return get_match_amount () == 1 && kind == Kind::MetaVar; + } + + const MatchedFragment get_single_fragment () const + { + rust_assert (is_single_fragment ()); + + return fragments[0]; + } + + const Kind &get_kind () const { return kind; } + +private: + /** + * Fragments matched `match_amount` times. This can be an empty vector + * in case having zero matches is allowed (i.e ? or * operators) + */ + std::vector fragments; + Kind kind; +}; + +class SubstitutionScope +{ +public: + SubstitutionScope () : stack () {} + + void push () { stack.push_back ({}); } + + std::map pop () + { + auto top = stack.back (); + stack.pop_back (); + return top; + } + + std::map &peek () + { + return stack.back (); + } + + /** + * Insert a new matched metavar into the current substitution map + */ + void insert_metavar (MatchedFragment fragment) + { + auto ¤t_map = stack.back (); + auto it = current_map.find (fragment.fragment_ident); + + if (it == current_map.end ()) + current_map.insert ({fragment.fragment_ident, + MatchedFragmentContainer::metavar (fragment)}); + else + gcc_unreachable (); + } + + /** + * Append a new matched fragment to a repetition into the current substitution + * map + */ + void append_fragment (MatchedFragment fragment) + { + auto ¤t_map = stack.back (); + auto it = current_map.find (fragment.fragment_ident); + + if (it == current_map.end ()) + current_map.insert ( + {fragment.fragment_ident, MatchedFragmentContainer ({fragment})}); + else + it->second.add_fragment (fragment); + } + + void insert_matches (std::string key, MatchedFragmentContainer matches) + { + auto ¤t_map = stack.back (); + auto it = current_map.find (key); + rust_assert (it == current_map.end ()); + + current_map.insert ({key, matches}); + } + +private: + std::vector> stack; +}; + +// Object used to store shared data (between functions) for macro expansion. +struct MacroExpander +{ + enum ContextType + { + ITEM, + BLOCK, + EXTERN, + TYPE, + TRAIT, + IMPL, + TRAIT_IMPL, + }; + + ExpansionCfg cfg; + unsigned int expansion_depth = 0; + + MacroExpander (AST::Crate &crate, ExpansionCfg cfg, Session &session) + : cfg (cfg), crate (crate), session (session), + sub_stack (SubstitutionScope ()), + expanded_fragment (AST::ASTFragment::create_error ()), + resolver (Resolver::Resolver::get ()), + mappings (Analysis::Mappings::get ()) + {} + + ~MacroExpander () = default; + + // Expands all macros in the crate passed in. + void expand_crate (); + + /* Expands a macro invocation - possibly make both + * have similar duck-typed interface and use templates?*/ + // should this be public or private? + void expand_invoc (AST::MacroInvocation &invoc, bool has_semicolon); + + // Expands a single declarative macro. + AST::ASTFragment expand_decl_macro (Location locus, + AST::MacroInvocData &invoc, + AST::MacroRulesDefinition &rules_def, + bool semicolon); + + void expand_cfg_attrs (AST::AttrVec &attrs); + bool fails_cfg (const AST::AttrVec &attr) const; + bool fails_cfg_with_expand (AST::AttrVec &attrs) const; + + bool depth_exceeds_recursion_limit () const; + + bool try_match_rule (AST::MacroRule &match_rule, + AST::DelimTokenTree &invoc_token_tree); + + AST::ASTFragment transcribe_rule ( + AST::MacroRule &match_rule, AST::DelimTokenTree &invoc_token_tree, + std::map &matched_fragments, + bool semicolon, ContextType ctx); + + bool match_fragment (Parser &parser, + AST::MacroMatchFragment &fragment); + + bool match_token (Parser &parser, AST::Token &token); + + bool match_repetition (Parser &parser, + AST::MacroMatchRepetition &rep); + + bool match_matcher (Parser &parser, + AST::MacroMatcher &matcher); + + /** + * Match any amount of matches + * + * @param parser Parser to use for matching + * @param rep Repetition to try and match + * @param match_amount Reference in which to store the ammount of succesful + * and valid matches + * + * @param lo_bound Lower bound of the matcher. When specified, the matcher + * will only succeed if it parses at *least* `lo_bound` fragments. If + * unspecified, the matcher could succeed when parsing 0 fragments. + * + * @param hi_bound Higher bound of the matcher. When specified, the matcher + * will only succeed if it parses *less than* `hi_bound` fragments. If + * unspecified, the matcher could succeed when parsing an infinity of + * fragments. + * + * @return true if matching was successful and within the given limits, false + * otherwise + */ + bool match_n_matches (Parser &parser, + AST::MacroMatchRepetition &rep, size_t &match_amount, + size_t lo_bound = 0, size_t hi_bound = 0); + + void push_context (ContextType t) { context.push_back (t); } + + ContextType pop_context () + { + rust_assert (!context.empty ()); + + ContextType t = context.back (); + context.pop_back (); + + return t; + } + + ContextType peek_context () { return context.back (); } + + void set_expanded_fragment (AST::ASTFragment &&fragment) + { + expanded_fragment = std::move (fragment); + } + + AST::ASTFragment take_expanded_fragment (AST::ASTVisitor &vis) + { + AST::ASTFragment old_fragment = std::move (expanded_fragment); + auto accumulator = std::vector (); + expanded_fragment = AST::ASTFragment::create_error (); + + for (auto &node : old_fragment.get_nodes ()) + { + expansion_depth++; + node.accept_vis (vis); + // we'll decide the next move according to the outcome of the macro + // expansion + if (expanded_fragment.is_error ()) + accumulator.push_back (node); // if expansion fails, there might be a + // non-macro expression we need to keep + else + { + // if expansion succeeded, then we need to merge the fragment with + // the contents in the accumulator, so that our final expansion + // result will contain non-macro nodes as it should + auto new_nodes = expanded_fragment.get_nodes (); + std::move (new_nodes.begin (), new_nodes.end (), + std::back_inserter (accumulator)); + expanded_fragment = AST::ASTFragment (accumulator); + } + expansion_depth--; + } + + return old_fragment; + } + +private: + AST::Crate &crate; + Session &session; + SubstitutionScope sub_stack; + std::vector context; + AST::ASTFragment expanded_fragment; + +public: + Resolver::Resolver *resolver; + Analysis::Mappings *mappings; +}; + +} // namespace Rust + +#endif diff --git a/gcc/rust/expand/rust-macro-invoc-lexer.cc b/gcc/rust/expand/rust-macro-invoc-lexer.cc new file mode 100644 index 00000000000..8a43d29e0d1 --- /dev/null +++ b/gcc/rust/expand/rust-macro-invoc-lexer.cc @@ -0,0 +1,29 @@ +#include "rust-macro-invoc-lexer.h" + +namespace Rust { + +const_TokenPtr +MacroInvocLexer::peek_token (int n) +{ + if ((offs + n) >= token_stream.size ()) + return Token::make (END_OF_FILE, Location ()); + + return token_stream.at (offs + n)->get_tok_ptr (); +} + +// Advances current token to n + 1 tokens ahead of current position. +void +MacroInvocLexer::skip_token (int n) +{ + offs += (n + 1); +} + +void +MacroInvocLexer::split_current_token (TokenId new_left __attribute__ ((unused)), + TokenId new_right + __attribute__ ((unused))) +{ + // FIXME + gcc_unreachable (); +} +} // namespace Rust diff --git a/gcc/rust/expand/rust-macro-invoc-lexer.h b/gcc/rust/expand/rust-macro-invoc-lexer.h new file mode 100644 index 00000000000..0fd4554d02f --- /dev/null +++ b/gcc/rust/expand/rust-macro-invoc-lexer.h @@ -0,0 +1,64 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#ifndef RUST_MACRO_INVOC_LEXER_H +#define RUST_MACRO_INVOC_LEXER_H + +#include "rust-ast.h" + +namespace Rust { +class MacroInvocLexer +{ +public: + MacroInvocLexer (std::vector> stream) + : offs (0), token_stream (std::move (stream)) + {} + + // Returns token n tokens ahead of current position. + const_TokenPtr peek_token (int n); + + // Peeks the current token. + const_TokenPtr peek_token () { return peek_token (0); } + + // Advances current token to n + 1 tokens ahead of current position. + void skip_token (int n); + + // Skips the current token. + void skip_token () { skip_token (0); } + + // Splits the current token into two. Intended for use with nested generics + // closes (i.e. T> where >> is wrongly lexed as one token). Note that + // this will only work with "simple" tokens like punctuation. + void split_current_token (TokenId new_left, TokenId new_right); + + std::string get_filename () const + { + // FIXME + gcc_unreachable (); + return "FIXME"; + } + + size_t get_offs () const { return offs; } + +private: + size_t offs; + std::vector> token_stream; +}; +} // namespace Rust + +#endif // RUST_MACRO_INVOC_LEXER_H diff --git a/gcc/rust/expand/rust-macro-substitute-ctx.cc b/gcc/rust/expand/rust-macro-substitute-ctx.cc new file mode 100644 index 00000000000..9592d2d2a9e --- /dev/null +++ b/gcc/rust/expand/rust-macro-substitute-ctx.cc @@ -0,0 +1,312 @@ +#include "rust-macro-substitute-ctx.h" + +namespace Rust { + +std::vector> +SubstituteCtx::substitute_metavar (std::unique_ptr &metavar) +{ + auto metavar_name = metavar->get_str (); + + std::vector> expanded; + auto it = fragments.find (metavar_name); + if (it == fragments.end ()) + { + // Return a copy of the original token + expanded.push_back (metavar->clone_token ()); + } + else + { + // If we are expanding a metavar which has a lof of matches, we are + // currently expanding a repetition metavar - not a simple metavar. We + // need to error out and inform the user. + // Associated test case for an example: compile/macro-issue1224.rs + if (it->second.get_match_amount () != 1) + { + rust_error_at (metavar->get_locus (), + "metavariable is still repeating at this depth"); + rust_inform ( + metavar->get_locus (), + "you probably forgot the repetition operator: %<%s%s%s%>", "$(", + metavar->as_string ().c_str (), ")*"); + return expanded; + } + + // We only care about the vector when expanding repetitions. + // Just access the first element of the vector. + auto &frag = it->second.get_single_fragment (); + for (size_t offs = frag.token_offset_begin; offs < frag.token_offset_end; + offs++) + { + auto &tok = input.at (offs); + expanded.push_back (tok->clone_token ()); + } + } + + return expanded; +} + +bool +SubstituteCtx::check_repetition_amount (size_t pattern_start, + size_t pattern_end, + size_t &expected_repetition_amount) +{ + bool first_fragment_found = false; + bool is_valid = true; + + for (size_t i = pattern_start; i < pattern_end; i++) + { + if (macro.at (i)->get_id () == DOLLAR_SIGN) + { + auto &frag_token = macro.at (i + 1); + if (frag_token->get_id () == IDENTIFIER) + { + auto it = fragments.find (frag_token->get_str ()); + if (it == fragments.end ()) + { + // If the repetition is not anything we know (ie no declared + // metavars, or metavars which aren't present in the + // fragment), we can just error out. No need to paste the + // tokens as if nothing had happened. + rust_error_at (frag_token->get_locus (), + "metavar %s used in repetition does not exist", + frag_token->get_str ().c_str ()); + + is_valid = false; + } + + auto &fragment = it->second; + + size_t repeat_amount = fragment.get_match_amount (); + if (!first_fragment_found) + { + first_fragment_found = true; + expected_repetition_amount = repeat_amount; + } + else + { + if (repeat_amount != expected_repetition_amount + && !fragment.is_single_fragment ()) + { + rust_error_at ( + frag_token->get_locus (), + "different amount of matches used in merged " + "repetitions: expected %lu, got %lu", + (unsigned long) expected_repetition_amount, + (unsigned long) repeat_amount); + is_valid = false; + } + } + } + } + } + + return is_valid; +} + +std::vector> +SubstituteCtx::substitute_repetition ( + size_t pattern_start, size_t pattern_end, + std::unique_ptr separator_token) +{ + rust_assert (pattern_end < macro.size ()); + + size_t repeat_amount = 0; + if (!check_repetition_amount (pattern_start, pattern_end, repeat_amount)) + return {}; + + rust_debug ("repetition amount to use: %lu", (unsigned long) repeat_amount); + std::vector> expanded; + std::vector> new_macro; + + // We want to generate a "new macro" to substitute with. This new macro + // should contain only the tokens inside the pattern + for (size_t tok_idx = pattern_start; tok_idx < pattern_end; tok_idx++) + new_macro.emplace_back (macro.at (tok_idx)->clone_token ()); + + // Then, we want to create a subset of the matches so that + // `substitute_tokens()` can only see one fragment per metavar. Let's say we + // have the following user input: (1 145 'h') + // on the following match arm: ($($lit:literal)*) + // which causes the following matches: { "lit": [1, 145, 'h'] } + // + // The pattern (new_macro) is `$lit:literal` + // The first time we expand it, we want $lit to have the following token: 1 + // The second time, 145 + // The third and final time, 'h' + // + // In order to do so we must create "sub maps", which only contain parts of + // the original matches + // sub-maps: [ { "lit": 1 }, { "lit": 145 }, { "lit": 'h' } ] + // + // and give them to `substitute_tokens` one by one. + + for (size_t i = 0; i < repeat_amount; i++) + { + std::map sub_map; + for (auto &kv_match : fragments) + { + MatchedFragment sub_fragment; + + // FIXME: Hack: If a fragment is not repeated, how does it fit in the + // submap? Do we really want to expand it? Is this normal behavior? + if (kv_match.second.is_single_fragment ()) + sub_fragment = kv_match.second.get_single_fragment (); + else + sub_fragment = kv_match.second.get_fragments ()[i]; + + sub_map.insert ( + {kv_match.first, MatchedFragmentContainer::metavar (sub_fragment)}); + } + + auto substitute_context = SubstituteCtx (input, new_macro, sub_map); + auto new_tokens = substitute_context.substitute_tokens (); + + // Skip the first repetition, but add the separator to the expanded + // tokens if it is present + if (i != 0 && separator_token) + expanded.emplace_back (separator_token->clone_token ()); + + for (auto &new_token : new_tokens) + expanded.emplace_back (new_token->clone_token ()); + } + + // FIXME: We also need to make sure that all subsequent fragments + // contain the same amount of repetitions as the first one + + return expanded; +} + +static bool +is_rep_op (std::unique_ptr &tok) +{ + auto id = tok->get_id (); + return id == QUESTION_MARK || id == ASTERISK || id == PLUS; +} + +std::pair>, size_t> +SubstituteCtx::substitute_token (size_t token_idx) +{ + auto &token = macro.at (token_idx); + switch (token->get_id ()) + { + case IDENTIFIER: + rust_debug ("expanding metavar: %s", token->get_str ().c_str ()); + return {substitute_metavar (token), 1}; + case LEFT_PAREN: { + // We need to parse up until the closing delimiter and expand this + // fragment->n times. + rust_debug ("expanding repetition"); + + // We're in a context where macro repetitions have already been + // parsed and validated: This means that + // 1/ There will be no delimiters as that is an error + // 2/ There are no fragment specifiers anymore, which prevents us + // from reusing parser functions. + // + // Repetition patterns are also special in that they cannot contain + // "rogue" delimiters: For example, this is invalid, as they are + // parsed as MacroMatches and must contain a correct amount of + // delimiters. + // `$($e:expr ) )` + // ^ rogue closing parenthesis + // + // With all of that in mind, we can simply skip ahead from one + // parenthesis to the other to find the pattern to expand. Of course, + // pairs of delimiters, including parentheses, are allowed. + // `$($e:expr ( ) )` + // Parentheses are the sole delimiter for which we need a special + // behavior since they delimit the repetition pattern + + size_t pattern_start = token_idx + 1; + size_t pattern_end = pattern_start; + auto parentheses_stack = 0; + for (size_t idx = pattern_start; idx < macro.size (); idx++) + { + if (macro.at (idx)->get_id () == LEFT_PAREN) + { + parentheses_stack++; + } + else if (macro.at (idx)->get_id () == RIGHT_PAREN) + { + if (parentheses_stack == 0) + { + pattern_end = idx; + break; + } + parentheses_stack--; + } + } + + // Unreachable case, but let's make sure we don't ever run into it + rust_assert (pattern_end != pattern_start); + + std::unique_ptr separator_token = nullptr; + if (pattern_end + 1 <= macro.size ()) + { + auto &post_pattern_token = macro.at (pattern_end + 1); + if (!is_rep_op (post_pattern_token)) + separator_token = post_pattern_token->clone_token (); + } + + // Amount of tokens to skip + auto to_skip = 0; + // Parentheses + to_skip += 2; + // Repetition operator + to_skip += 1; + // Separator + if (separator_token) + to_skip += 1; + + return {substitute_repetition (pattern_start, pattern_end, + std::move (separator_token)), + pattern_end - pattern_start + to_skip}; + } + // TODO: We need to check if the $ was alone. In that case, do + // not error out: Simply act as if there was an empty identifier + // with no associated fragment and paste the dollar sign in the + // transcription. Unsure how to do that since we always have at + // least the closing curly brace after an empty $... + default: + rust_error_at (token->get_locus (), + "unexpected token in macro transcribe: expected " + "%<(%> or identifier after %<$%>, got %<%s%>", + get_token_description (token->get_id ())); + } + + // FIXME: gcc_unreachable() error case? + return {std::vector> (), 0}; +} + +std::vector> +SubstituteCtx::substitute_tokens () +{ + std::vector> replaced_tokens; + rust_debug ("expanding tokens"); + + for (size_t i = 0; i < macro.size (); i++) + { + auto &tok = macro.at (i); + if (tok->get_id () == DOLLAR_SIGN) + { + // Aaaaah, if only we had C++17 :) + // auto [expanded, tok_to_skip] = ... + auto p = substitute_token (i + 1); + auto expanded = std::move (p.first); + auto tok_to_skip = p.second; + + i += tok_to_skip; + + for (auto &token : expanded) + replaced_tokens.emplace_back (token->clone_token ()); + } + else + { + replaced_tokens.emplace_back (tok->clone_token ()); + } + } + + return replaced_tokens; +} + +} // namespace Rust diff --git a/gcc/rust/expand/rust-macro-substitute-ctx.h b/gcc/rust/expand/rust-macro-substitute-ctx.h new file mode 100644 index 00000000000..81dcab7643b --- /dev/null +++ b/gcc/rust/expand/rust-macro-substitute-ctx.h @@ -0,0 +1,93 @@ +// Copyright (C) 2020-2022 Free Software Foundation, Inc. + +// This file is part of GCC. + +// GCC is free software; you can redistribute it and/or modify it under +// the terms of the GNU General Public License as published by the Free +// Software Foundation; either version 3, or (at your option) any later +// version. + +// GCC is distributed in the hope that it will be useful, but WITHOUT ANY +// WARRANTY; without even the implied warranty of MERCHANTABILITY or +// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +// for more details. + +// You should have received a copy of the GNU General Public License +// along with GCC; see the file COPYING3. If not see +// . + +#include "rust-ast.h" +#include "rust-macro-expand.h" + +namespace Rust { +class SubstituteCtx +{ + std::vector> &input; + std::vector> ¯o; + std::map &fragments; + + /** + * Find the repetition amount to use when expanding a repetition, and + * check that all fragments used respect that repetition amount + * + * @param pattern_start Start of the repetition pattern + * @param pattern_end End of the repetition pattern + * @param repeat_amount Reference to fill with the matched repetition amount + */ + bool check_repetition_amount (size_t pattern_start, size_t pattern_end, + size_t &repeat_amount); + +public: + SubstituteCtx (std::vector> &input, + std::vector> ¯o, + std::map &fragments) + : input (input), macro (macro), fragments (fragments) + {} + + /** + * Substitute a metavariable by its given fragment in a transcribing context, + * i.e. replacing $var with the associated fragment. + * + * @param metavar Metavariable to try and replace + * + * @return A token containing the associated fragment expanded into tokens if + * any, or the cloned token if no fragment was associated + */ + std::vector> + substitute_metavar (std::unique_ptr &metavar); + + /** + * Substitute a macro repetition by its given fragments + * + * @param pattern_start Start index of the pattern tokens + * @param pattern_end End index of the patterns tokens + * @param separator Optional separator to include when expanding tokens + * + * @return A vector containing the repeated pattern + */ + std::vector> + substitute_repetition (size_t pattern_start, size_t pattern_end, + std::unique_ptr separator); + + /** + * Substitute a given token by its appropriate representation + * + * @param token_idx Current token to try and substitute + * + * @return A token containing the associated fragment expanded into tokens if + * any, or the cloned token if no fragment was associated, as well as the + * amount of tokens that should be skipped before the next invocation. Since + * this function may consume more than just one token, it is important to skip + * ahead of the input to avoid mis-substitutions + */ + std::pair>, size_t> + substitute_token (size_t token_idx); + + /** + * Substitute all tokens by their appropriate representation + * + * @return A vector containing the substituted tokens + */ + std::vector> substitute_tokens (); +}; +} // namespace Rust