From patchwork Tue Nov 7 18:18:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Burgess X-Patchwork-Id: 79350 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4B4153858035 for ; Tue, 7 Nov 2023 18:19:26 +0000 (GMT) X-Original-To: gdb-patches@sourceware.org Delivered-To: gdb-patches@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 2E2913858CDB for ; Tue, 7 Nov 2023 18:18:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2E2913858CDB Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2E2913858CDB Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699381140; cv=none; b=UmArZDthZr8vmgSbNMKjuy6HK1jj+WM8w9F7iQq34u/ykQOY3Ozi2UuAo+kJ2f43pHw560z62WZdqCCNOm2vt/g4w3bvY/iRk4fA0vkauJrwGLAeAtK3rw+3qnxzCPANGLDCEi2js76fyHkqZwRekpMN018Ax6d7S1aaFPlVelg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699381140; c=relaxed/simple; bh=5OuHH+df8HS2j7pA6fsjcN6Rmx6v8WaZX6f8zBV9kuU=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=mQLztIpUuM0KEkeVdtuHWJl2QjLHXViJXZGXKPw/fO5pyoKxEXLI+/ALl9sjYXxTg1LUgfEpJCiY5fFcd11qYBkyFoPyl4RDKxoNLmoGuDzcdMYwiRAMP/5TCUspBWTHxGqlBJI02JaODn5H7cqjujdwbQ9JrY05iZ2gO9oUnXA= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699381138; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n1DHofxdpg/JTmZfLd9MeFN44Srikmbrn2VIAnnJh0M=; b=SFGw3R/zJxIZGoJgU+IH2UZKmOwBDrmoApq+CKB165kCwkk4g8ak4Kc5qXwEUiJfhe9Ewf mQjuigCgQy8xSU9cKgUz1OAjCb787bMSqM6817AXg2gG3NG+h0ot4IKm35pI4b7waLh7bW LqhYjtFmywZjVlUqezTZBI6vvJDnG4g= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-90--7K6TDfVO0iwKI7AbD4Qog-1; Tue, 07 Nov 2023 13:18:57 -0500 X-MC-Unique: -7K6TDfVO0iwKI7AbD4Qog-1 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5082705f6dfso6002834e87.2 for ; Tue, 07 Nov 2023 10:18:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699381135; x=1699985935; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n1DHofxdpg/JTmZfLd9MeFN44Srikmbrn2VIAnnJh0M=; b=W1kOFGZ5zljr6ID5pOWpyDSEa+SFE4YmUSlOSdfFA7tIBsEHaTd/JNJCj1MVlj5hV3 Y47o9b81XENP6wfxxTWH801wUan77WQ7WFXtzVRBL7kmYTpKAR5HjE3cfx1Cbq9amL+/ skKAHpiJCVdXRa4C5tjNVZN//S8h/bYY0P5P+rI2oAOmEmlZo/m2iuyk9FwouSvZqNh6 uuzvge0caDX67gbO+edQ/E7Zym5f/r0JxsQQrydXnIAhnz4L1Cx+Z/W/AjorhJy2WHYz 450a+vcSuHxdUkIbxSK7QKjk+XTgefQuu9xHJmMTxlED4BWvmnHs6Kde6w8hJm+MhJ6C r99w== X-Gm-Message-State: AOJu0YySSK9zacIYv6+ZCCsisTr2p9dU+uQR32eLI9o9e8XxJvpHPbAh kCCduqUfv2datlRL7Ouk8c5iRouYgwPvrPYSq+v4Nrvn3z81lP36hsr7h/46b38Kg5JGImHWSAt dE3yfb6Vc+KP03OWLFAor1VVWXWxEMzr+yscmrESHA6Jr8HzMLoAr8xgoKLqmlJrVgqdCyxof3K REk1JeOA== X-Received: by 2002:a05:6512:3092:b0:503:8fa:da21 with SMTP id z18-20020a056512309200b0050308fada21mr31955416lfd.43.1699381135285; Tue, 07 Nov 2023 10:18:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IFKlCyWH2Aa+9DCggsuu7m9ZKP1FtdtL7woe91bQ8NgrNnoP5Q64VGWP5UKPZF00PuFPLFoog== X-Received: by 2002:a05:6512:3092:b0:503:8fa:da21 with SMTP id z18-20020a056512309200b0050308fada21mr31955399lfd.43.1699381134888; Tue, 07 Nov 2023 10:18:54 -0800 (PST) Received: from localhost (105.226.159.143.dyn.plus.net. [143.159.226.105]) by smtp.gmail.com with ESMTPSA id m17-20020a056000181100b0032da6f17ffdsm2995950wrh.38.2023.11.07.10.18.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:18:54 -0800 (PST) From: Andrew Burgess To: gdb-patches@sourceware.org Cc: Andrew Burgess Subject: [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Date: Tue, 7 Nov 2023 18:18:48 +0000 Message-Id: <047bb72318595c2c3e2435bc0b8ff062515c28e3.1699381067.git.aburgess@redhat.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: References: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gdb-patches-bounces+patchwork=sourceware.org@sourceware.org I have been making more use of the make-check-all.sh script to run tests against all boards. But one thing is pretty annoying. When a test fails on some random board, I have to run make-check-all.sh with --verbose and --dry-run in order to see what RUNTESTFLAGS I should be using. I always run with --keep-results on, so, in this commit, I propose that, when --keep-results is on the 'make check' command will be written out to a file within the stored results directory, like: check-all/BOARD_NAME/make-check.sh then, if I want to rerun a test, I can just: sh check-all/BOARD_NAME/make-check.sh and the test will be re-run for me. --- gdb/testsuite/make-check-all.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh index 7d8adb58e87..6d67567758f 100755 --- a/gdb/testsuite/make-check-all.sh +++ b/gdb/testsuite/make-check-all.sh @@ -186,6 +186,10 @@ do_tests () mkdir -p "$dir" cp gdb.sum gdb.log "$dir" + + # Record the 'make check' command to enable easy re-running. + echo "make check RUNTESTFLAGS=\"${rtf[*]} ${tests[*]}\"" \ + > "$dir/make-check.sh" fi } From patchwork Tue Nov 7 18:18:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Burgess X-Patchwork-Id: 79349 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D32EE3857701 for ; Tue, 7 Nov 2023 18:19:16 +0000 (GMT) X-Original-To: gdb-patches@sourceware.org Delivered-To: gdb-patches@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 4D9B13858C33 for ; Tue, 7 Nov 2023 18:19:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4D9B13858C33 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4D9B13858C33 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699381143; cv=none; b=vwpMaULB7HZXyTmjErMfBUxADVd9a1Ny8Jx1ReftCBtk9ontFtjXqxdFFtbek1eXKKcZudwKH/DCw++15BL94bcHDNqk5j46/Q7qUCrisvmO9s4l/LkOHX/wGTTJeyg14BhPQ1nGOWAu3nKbrbfAi2N4QtKpMyndnDGmkQxVRkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699381143; c=relaxed/simple; bh=SC7Cg5t+J4bPfTOhCEJ21haIPajUoDTPLwgwVaoXDZY=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=jD7a3FSxh2/xt/rtmpFpdjsJo2z2NcMCGR+8enw1bmIW/SYy6IhPumvsfywbten2z1796GtvSeBE09P87hhZdzK8AfIZg+tF4HMz9H5btjXgErPohyxsTs4M98x1nlqFGHytlGb72zD/xDTJ1IcvIzjSJ/oROX7yViaR8LorCsg= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699381141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nUZEKBXddizE+qYKYBx6GSCegi9REdzFbGY/5oSF4kM=; b=MP3F9TZWxmoaaicg+ON3bWdww0RZ19Exhd1JsOMJe/uj3UEhreIFSH2J5tY1H/Du2Ew+yw XlYB0wOx221+NCwazCyi6MJutIVzKyQVhursRB3eNnk+aEPbC64JhLhqW7kRbCYV/RUIWT MrHE/GPF4Z6ZAyDgApMmPAGqf0LhydQ= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-638-oXUacG9YMNWPQPgRHqb8gw-1; Tue, 07 Nov 2023 13:18:59 -0500 X-MC-Unique: oXUacG9YMNWPQPgRHqb8gw-1 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-2c50257772bso55872431fa.3 for ; Tue, 07 Nov 2023 10:18:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699381137; x=1699985937; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nUZEKBXddizE+qYKYBx6GSCegi9REdzFbGY/5oSF4kM=; b=WLd3snsod37UKYmOJ9hvbMG06+kjSo+KIi1Zwc1lQNMLAc2MurpF8UsIwJcIA1uUvP OjiZ9jjJqJyySQuwpshqfqiKEvwSFRSZaeu4HIlrBztuHR/7RmfKHzbwkdZi0VQwF5zM LSKpk5vqU9A2Vmre5a5jLazAjfuV6BG8DTIIuQ2uKoSp/MLm8Sk3wc3x8ASYbBcfdmW0 ikfD2ebdx9M0QywtIJFn0LDO7UQak0rqOEHii7wM8gB9iTEnEyr+XymYjdi8oKSmEWEA fNlVYxdD9F0NrCvm8pHRTCj7f0Z44FL6DYhYlshGyDJUDKGJm67VkTHQiwj47tSOeONK Iibw== X-Gm-Message-State: AOJu0Yxn4Yp5udZUQitFOWPuXN2H/FYV6TzTxrZPt25pT2DgL/nkdpVi tfFS0y1G3YPzRUtBKjw8GzhJMEvhvM7EcmJIyvlD0wUEJ/0N+HJhm4mkLrlM7q5kXRBYUaluMh9 PIYZMnJHx5o+GxER8kAtSK8rbTBFnNBgfjXdDwXsogPpOuUWAfxrSiy/F4Y496UYbX6h/BY/fSG 3SifmEjw== X-Received: by 2002:a05:651c:54a:b0:2c7:fa6:718c with SMTP id q10-20020a05651c054a00b002c70fa6718cmr10666833ljp.9.1699381136919; Tue, 07 Nov 2023 10:18:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IGBc+kEFOFfDfh2zSu4oKUZlry9dzGur69mlyUvwvALPrbDFWIBT/FU5VyUA3HzhzGQusOUEg== X-Received: by 2002:a05:651c:54a:b0:2c7:fa6:718c with SMTP id q10-20020a05651c054a00b002c70fa6718cmr10666811ljp.9.1699381136497; Tue, 07 Nov 2023 10:18:56 -0800 (PST) Received: from localhost (105.226.159.143.dyn.plus.net. [143.159.226.105]) by smtp.gmail.com with ESMTPSA id n26-20020a1c721a000000b0040849ce7116sm16193180wmc.43.2023.11.07.10.18.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:18:56 -0800 (PST) From: Andrew Burgess To: gdb-patches@sourceware.org Cc: Andrew Burgess Subject: [PATCH 2/2] gdb/testsuite: add a new check-all-boards target Date: Tue, 7 Nov 2023 18:18:49 +0000 Message-Id: X-Mailer: git-send-email 2.25.4 In-Reply-To: References: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gdb-patches-bounces+patchwork=sourceware.org@sourceware.org The make-check-all.sh script (gdb/testsuite/make-check-all.sh) is great, it makes it super easy to run some test(s) using all the available board files. This commit aims to make this script even easier to access by adding a check-all-boards target to the GDB Makefile. This new target checks for (and requires) a number of environment variables, so the target should be used like this: make check-all-boards GDB_TARGET_USERNAME=remote-target \ GDB_HOST_USERNAME=remote-host \ TESTS="gdb.base/break.exp" Where GDB_TARGET_USERNAME and GDB_HOST_USERNAME are the user names that should be passed to the make-check-all.sh --target-user and --host-user command line options respectively. My personal intention is to set these variables in my environment, so all I'll need to do is: make check-all-boards TESTS="gdb.base/break.exp" The make rule always passes --keep-results to the make-check-all.sh script, as I find that the most useful. It's super frustrating to run the tests and realise you forgot that option and the results have been discarded. --- gdb/Makefile.in | 8 ++++++++ gdb/testsuite/Makefile.in | 19 ++++++++++++++++++ gdb/testsuite/README | 42 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 69 insertions(+) diff --git a/gdb/Makefile.in b/gdb/Makefile.in index 9c0a0bff2cd..491003c7543 100644 --- a/gdb/Makefile.in +++ b/gdb/Makefile.in @@ -1971,6 +1971,14 @@ check-parallel: force $(MAKE) $(TARGET_FLAGS_TO_PASS) check-parallel; \ else true; fi +check-all-boards: force + @if [ -f testsuite/Makefile ]; then \ + rootme=`pwd`; export rootme; \ + rootsrc=`cd $(srcdir); pwd`; export rootsrc; \ + cd testsuite; \ + $(MAKE) $(TARGET_FLAGS_TO_PASS) check-all-boards; \ + else true; fi + # The idea is to parallelize testing of multilibs, for example: # make -j3 check//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu} # will run 3 concurrent sessions of check, eventually testing all 10 diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in index 3fc9a907f5b..3488e6acc48 100644 --- a/gdb/testsuite/Makefile.in +++ b/gdb/testsuite/Makefile.in @@ -368,6 +368,25 @@ check-perf: all $(abs_builddir)/site.exp @if test ! -d gdb.perf; then mkdir gdb.perf; fi $(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFTEST_MODE=$(GDB_PERFTEST_MODE) $(RUNTESTFLAGS) $(TIMESTAMP) + +check-all-boards: all $(abs_builddir)/site.exp + @if [ -z "$(GDB_TARGET_USERNAME)" ]; then \ + echo "Must define GDB_TARGET_USERNAME"; \ + false; \ + fi + @if [ -z "$(GDB_HOST_USERNAME)" ]; then \ + echo "Must define GDB_HOST_USERNAME"; \ + false; \ + fi + @if [ -z "$(TESTS)" ]; then \ + echo "Must define TESTS"; \ + false; \ + fi + ${abs_srcdir}/make-check-all.sh --keep-results \ + --host-user "$(GDB_HOST_USERNAME)" \ + --target-user "$(GDB_TARGET_USERNAME)" \ + "$(TESTS)" + force:; clean mostlyclean: diff --git a/gdb/testsuite/README b/gdb/testsuite/README index 330a385f349..5ad0293649e 100644 --- a/gdb/testsuite/README +++ b/gdb/testsuite/README @@ -328,6 +328,48 @@ For example, to turn on gdbserver debugging, you can do: make check GDBSERVER_DEBUG="debug,replay" +GDB_TARGET_USERNAME +GDB_HOST_USERNAME + +These settings are only used with the check-all-boards target, and +should be the usernames of two separate users on the local machine, +both of which the current user can be ssh to without a password. + +These users will be used when by board files that simulate remote +targets by switching to a different user on the same machine. These +users will have random files copied into their $HOME directories, so +it is a good idea to setup new users just for this purpose. + +Testing All Simple Boards +************************* + +There are a number of boards that ship with GDB that simulate common +debug scenarios. For example by sshing to a different user on the +local machine and running gdbserver as this alternative user we aim to +simulate a true remote debug experience. + +There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which +can be used to run a defined set of tests using all of the available +simple board files. Support for using this script is also included in +GDB's makefile, and can be used as: + + make check-all-boards GDB_TARGET_USERNAME=remote-target \ + GDB_HOST_USERNAME=remote-host \ + TESTS="gdb.base/break.exp" + +The 'remote-target' and 'remote-host' can be replaced with any user +names on the local machine, the only requirements are that the current +user must be able to ssh to these users without a password, and these +users must be happy to have arbitrary files copied into their $HOME +directory. Ideally, these users would be setup just for GDB testing. + +The check-all-boards target requires that TESTS be defined, though it +is fine to include multiple tests. + +There results are preserved and can be found in +gdb/testsuite/check-all/, within this directory the results are split +by the board file used. + Architecture-specific Parameters ******************************