From patchwork Fri Sep 22 23:42:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steve Ellcey X-Patchwork-Id: 23101 Received: (qmail 83142 invoked by alias); 22 Sep 2017 23:42:23 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 83133 invoked by uid 89); 22 Sep 2017 23:42:22 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-25.3 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=H*M:174, H*MI:174, everytime, x X-HELO: NAM01-SN1-obe.outbound.protection.outlook.com Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Steve.Ellcey@cavium.com; Message-ID: <1506123735.2286.174.camel@cavium.com> Subject: [RFC][PATCH] Rerun benchmarks only if *.out file does not exist. From: Steve Ellcey Reply-To: sellcey@cavium.com To: libc-alpha Date: Fri, 22 Sep 2017 16:42:15 -0700 Mime-Version: 1.0 X-ClientProxiedBy: MWHPR02CA0046.namprd02.prod.outlook.com (10.164.133.35) To MWHPR07MB3550.namprd07.prod.outlook.com (10.164.192.139) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 14b64c19-9e1e-4515-0c05-08d502138f87 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(49563074)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:MWHPR07MB3550; X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3550; 3:/6lQnHDNmW9ejcdFcQJK9wSdOs/mVzL66pELzQJC1GXIjzYIlHXbzSCw3ImNNyhkdLy2yv76tH4+XEkpiV94M4r3o0YmmIr/l4yEmahZcwuX/VwaarHTsP+4IcjS/6XTglFpJssGv8yfJn9YtO6mFTN6KV5HfpIbx84bzYMgGK2/g1dQurOgr+91UfysxzWWiDVcwBPv/hadGQjMmfzCqhtIQWVPvBpZ/KpcUlk7PHsPUBPLniQVcGsfU6kDMQwy; 25:+49c4t0CfKGZTZovgZjVlyqGEVONVgA4guGeXTihAzZhcdH6WhUTzbEZhUd8eqo26Zplgxu6EPNA0/QhAIQ97BrdAFvcgOuh0ok7FGqsu3z8l+/8QjY19czrGFLCT3zQJSzlJGXy+slqgs4fw/oefVDSWxwoY6ifae646aWERV601Y1VX1K9VYp95MxxOpUdIjtGa6QblHvAbv0KlSYlSCNxSXXyBBQlC6mFXr+Tic4zRZjQR7ou96ydSxT59tXO4h3o+XG4u+CHH9Jy/kXFUUfIQqsl4v/hKxRsaxi2Fg9eO+O4nAHRW4L4XB6mbeV93ILrYVhG+Akig5TX8H78QA==; 31:TwIjkXYUDeQRhcp2mQZQpUoYDJuA8WwnN9aVA06NspREg4EeY85TxQa1zG4ZTgll5pZLn7QsXAgWWp1spfILQqeEX6m7UCa4ftn23WdfK5POSWo7lGN4uj/ZMoOi7ep8eVPVnm7r1ZgP6v8TzVjhRJcq6nA6zPSZkRda4ViSBrW4TSTIpHs+YM+GGnAinyV7iDr38HahMt0f/1Y57M7j4bm/ZH6llYfi6ixM6atfvoM= X-MS-TrafficTypeDiagnostic: MWHPR07MB3550: X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3550; 20:ZoR3wvNwGTrCV+eOpHOjQ3RdnuVMA5jwa4mudDyX/jzZNDn9STXulQJ3h7e7TjYbbGTbvx8nR7NXqaAfkXWXaUujDQnRDFMpxhO4BT4OwtU1j/sEGTriY2HIb89xiMsXjgl/PMZ5UEqacCc4XYf4NyCsg0NIu3WB81Gna/RiTojOWN5M2i6RFYXJhxIeB+qg6qPnSqPYXl/PULR4t4oQur3Wn3dzBlkuJt8Oak8j8FXEQyQU1WWIpFMtEtZMv5ubMNKZzORT9rGrAB9NFtgTH8qJl5mkxQLKBsRdUsu1UC8eeNYeaSHChXrn9V+3IGpR0oFM6RLFSv44AUW4co7rNXFHfa2GMLD3q+1l4YqhLVh2fvFMSvHMWfUd7kykYEI5dFeOosIddWXqElOyG6bbrMmZpL0PBu8e25trwnBAvr2zlII807cJOnLRyNqx/EaRtBPJl1JCOb9eGCW+6emOwugOAR1LH73tCg2qpQ0htRwIL4/KT7CptkFmfja46NTB; 4:R4aSoTTFVW/I1E0WC+tDq8Xcdq49GQTr1DlXd4SeOgCdnkeFhFl9u3NT78p10WwGTcWJ4vRRVodKyVFC95wC5Vy1LDAX9DjiMCizkHhIhIxNuFei2/F91EEOyo5+/3A59K+F5PfaWd1Hbqnnb5sZiRgsm7mjow2C+4bSn8fVXe3NVC+bwUoR7JRYoIYmcSrq7oA3t91nVfawSTa28HBRlrt8qomEsd04cxrNL837ewov9HQi+gqdnUB7I/2ni+aE X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(100000703101)(100105400095)(3002001)(10201501046)(6041248)(20161123560025)(20161123555025)(20161123558100)(20161123564025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR07MB3550; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR07MB3550; X-Forefront-PRVS: 0438F90F17 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(189002)(377424004)(199003)(16586007)(33646002)(5890100001)(6506006)(84326002)(72206003)(16526017)(5660300001)(53416004)(6486002)(103116003)(2476003)(69596002)(2906002)(8936002)(81156014)(81166006)(25786009)(97736004)(189998001)(316002)(8676002)(478600001)(50226002)(3450700001)(101416001)(43066003)(568964002)(6116002)(50986999)(68736007)(7736002)(3846002)(305945005)(106356001)(105586002)(66066001)(6512007)(6916009)(4610100001)(512874002)(53936002)(36756003)(99106002); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR07MB3550; H:sellcey-dt.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR07MB3550; 23:eFnuAxTEbRDrUB9cEooZenzqjhombfRFSsPzoGqxz?= =?us-ascii?Q?v17E7JdpK/4bl0d5md9MIWmM4mfhfIsNLyuV7rZEIdEp8bsJyqUDgsFgzO1S?= =?us-ascii?Q?U5AWxZm7NM4QuJiJ8z7oaORU9UFymjZE1JYMj67q5eQJucf1KvHy9e/m2Jwb?= =?us-ascii?Q?OiUhbGWHahVaIzp9GxrroQ36mdB4CWi85K0xf7AIA/nCBo+KrZJTw2KHHtcC?= =?us-ascii?Q?cw4XTXPQUU5cDm5nc9RrPDBQadQEkryzXe9xGvu+3Paeo7MCEI3/zjeeBMTn?= =?us-ascii?Q?ek6/yEp0GO1E9domILqgVxNXFPWQppIW+pfvqqjuYtbUq4oA3f+egvBozp1p?= =?us-ascii?Q?KrP4lSocOSQxbdnlyYzUxdQ1A+MvxXirzqDlfL03qfhn9m6j6ccQTuqjZkPI?= =?us-ascii?Q?Is2PCCaLIVwZgXHsnrDbJpv2f9Pg/x4vC6QAquoRUJlR98VH31+KyABnLkCx?= =?us-ascii?Q?FR9GkogmH7ibEW8UEw0lCrE+xsobs9vQs9sKCf0ckp+aB/b+9feh6yCoHn0Z?= =?us-ascii?Q?8O/nW+NA3qFzGZQ0vLTeGq98fzt95l/isn3bVZWD9kDGLXsNHI16DiHOaIxW?= =?us-ascii?Q?Dh0DYu3Eg7XOivOjeIJO5Ipoue7xRPwKMYnKc3KVZ8rRTQBHXe1g9n7M8e+0?= =?us-ascii?Q?cjJc6pUVrdfoS6lpIbPsA1IHzWHY3CWXCl5U2YeZBaCWjpPWbnjW+M212Q6y?= =?us-ascii?Q?sYaO1tgdmr1U75+vSCVfbmNGCMesG7JMg/yRTdqviTyf8jSHNYDC/bHrAovU?= =?us-ascii?Q?WCjZaP+xq1tbId039s0A37F0waEQV5pCBnl+JKQXFtP6XQywHfwDcV3DLJiO?= =?us-ascii?Q?t+otmL7/b4b/lwZQz81IaOS21itTsugSJzfh+/f5Em6LBAIWDTToAXyKcPtd?= =?us-ascii?Q?Wtb6/uc/mZhS2LDP/g2jyvr22tStnjunDNUEy7+9kpLNat5B5cjH5rw6TFvQ?= =?us-ascii?Q?cvpt7nRpFiihmbGCcsesxame/wl5jXzTIiYhYmvxNMCjxy8EQ7MXDzKUl8pI?= =?us-ascii?Q?0Pj/gYV+cgl7uP/9CmdiaOQWfT4JReM0CKUXeZ7qiFGwuYOZN/LLGCcIj4jO?= =?us-ascii?Q?Ev9jpvXIdl+n6dulcvi1ruZNkNNgRxjgxko8Fj2dTvr5wluwrJhhdgkAk/2U?= =?us-ascii?Q?VGdAWeXDAvp3wkfg+XYNNLJw7zm/RQHYaByA3znrZfk3e+zp/a1EbKcKfj1h?= =?us-ascii?Q?kKIKr+FnGwxzmw=3D?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3550; 6:fI/2pNBf74cDfMrfb8pTA0XtAnIo2mGr9F9TdZdEvPK1nxmdoY5AGyyJjJogG+VyNQfz62L2bk+suZhxrRcaWf9DxYsViIO5ocfrkF29kdxl8zynmQ91qqx837iSB6B7NDdi8Vkp0UP0EJ80MskbePr5knMzvWakJn/RjVjAEwwxiNAXp/XJ+hqtLYmPtC8LCqq/pz/5DVfc/GkR4ii1/4HfqISqmIiY6c79Sz9wTJqAVzG9vFOY3qit1bfBpKYPewvVY6TtBjIFlNB2aMh/+DZqt0ZKeZbdmStaUH5Uxzz0adZ3hrKxaW08Cx0U8xNtm2MkzihmAsuNauhF3yv4Mg==; 5:uuoD5mcrMRJDIxZ8T+dD/nYfxhTZbL7nABBekvVEYEBIk9rosSPbuS8tIkQq3B4kR1o9c+bf6pIXwbdDnn/ngYEM74QBhshdWmOBHvltSPCQVjjp++kgN0agD78x1U3NfotFUImQQJyfl4p++I29Tg==; 24:wP0OhNX7oO5t3Y/McpK8pILr46UQ4RLowBWvMLRIoYYJC19fR9fkO4xg48qb9RdK6z97QND86a8VD97z0bf5PWQ0Tx68aYJ6owTGWJLoInk=; 7:XTINLY6TmRrK3pnHa229ZI4+a0X5RMaeT9AJNNktLEMjUvnOXdkgi+gva6SMKoiPee6hUKQeDONY3Hhbz8CklvEs61W4LASfeHQ6nP6Xj+XnlTAf5hqKmBXTj0gUGgz4D+V0c0ytFda8kk/sAfVCsdXiJQojPYOJ7TUyhhDbqDALYjcNiKNWRIgvlTDhqgdW/Ojacmt8C97L26wQ/E6E+Vkn6ZItYunFIE85L+qnjBM= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2017 23:42:17.7482 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR07MB3550 While doing some benchmarking on glibc I noticed a difference between 'make check' and 'make bench'.  If you rerun 'make check' after having run it once already, it will only rerun tests where the *.out file does not exist.  This is useful (as explained in the glibc Wiki) because you can rerun one test by removing just that .out file and running 'make check'. With 'make bench' it reruns all the benchmarks each time you do 'make bench' even if the .out files exist.  I was thinking that it would be nice to have this follow the 'make check' behaviour, then if you only want to retest one test (because you are doing performance changes on one function), you can just remove that .out file and run 'make bench'.  Right now there does not seem to be a way to run one performance benchmark. Two questions: One, does this sound useful to other people?  And two, what is wrong with my patch?  This patch mostly works but the bench-memcpy, bench-memcpy-large, and bench-memcpy-random tests get rerun everytime even if the .out files for those tests already exist.  I don't know why. I am guessing is thas something to do with their common name prefix and the string manipulations done to test names in the Makefile but memset does not seem to have that problem. Steve Ellcey sellcey@cavium.com 2017-09-22  Steve Ellcey   * benchtests/Makefile (bench-set): Only run if *.out file does not exist. (bench-malloc): Likewise. (bench-func): Likewise. diff --git a/benchtests/Makefile b/benchtests/Makefile index 3acc39c..66b080e 100644 --- a/benchtests/Makefile +++ b/benchtests/Makefile @@ -168,15 +168,19 @@ endif bench-set: $(binaries-benchset) for run in $^; do \ - echo "Running $${run}"; \ - $(run-bench) > $${run}.out; \ + if test ! -f $${run}.out; then \ + echo "Running $${run}"; \ + $(run-bench) > $${run}.out; \ + fi \ done bench-malloc: $(binaries-bench-malloc) run=$(objpfx)bench-malloc-thread; \ for thr in 1 8 16 32; do \ - echo "Running $${run} $${thr}"; \ - $(run-bench) $${thr} > $${run}-$${thr}.out; \ + if test ! -f $${run}-$${thr}.out; then \ + echo "Running $${run} $${thr}"; \ + $(run-bench) $${thr} > $${run}-$${thr}.out; \ + fi \ done # Build and execute the benchmark functions. This target generates JSON @@ -184,25 +188,23 @@ bench-malloc: $(binaries-bench-malloc) # so one could even execute them individually and process it using any JSON # capable language or tool. bench-func: $(binaries-bench) - { timing_type=$$($(timing-type)); \ - echo "{\"timing_type\": \"$${timing_type}\","; \ - echo " \"functions\": {"; \ - for run in $^; do \ - if ! [ "x$${run}" = "x$<" ]; then \ - echo ","; \ - fi; \ - echo "Running $${run}" >&2; \ - $(run-bench) $(DETAILED_OPT); \ - done; \ - echo; \ - echo " }"; \ - echo "}"; } > $(objpfx)bench.out-tmp; \ - if [ -f $(objpfx)bench.out ]; then \ - mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \ - fi; \ - mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out - $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \ - scripts/benchout.schema.json + if test ! -f $(objpfx)bench.out; then \ + { timing_type=$$($(timing-type)); \ + echo "{\"timing_type\": \"$${timing_type}\","; \ + echo " \"functions\": {"; \ + for run in $^; do \ + if ! [ "x$${run}" = "x$<" ]; then \ + echo ","; \ + fi; \ + echo "Running $${run}" >&2; \ + $(run-bench) $(DETAILED_OPT); \ + done; \ + echo; \ + echo " }"; \ + echo "}"; } > $(objpfx)bench.out; \ + $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \ + scripts/benchout.schema.json; \ + fi $(timing-type) $(binaries-bench) $(binaries-benchset) \ $(binaries-bench-malloc): %: %.o $(objpfx)json-lib.o \