From patchwork Tue Mar 25 10:28:59 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siddhesh Poyarekar X-Patchwork-Id: 263 Return-Path: X-Original-To: siddhesh@wilcox.dreamhost.com Delivered-To: siddhesh@wilcox.dreamhost.com Received: from homiemail-mx22.g.dreamhost.com (caibbdcaabij.dreamhost.com [208.113.200.189]) by wilcox.dreamhost.com (Postfix) with ESMTP id 0C031360115 for ; Tue, 25 Mar 2014 03:28:34 -0700 (PDT) Received: by homiemail-mx22.g.dreamhost.com (Postfix, from userid 14307373) id AF5D55080869; Tue, 25 Mar 2014 03:28:34 -0700 (PDT) X-Original-To: glibc@patchwork.siddhesh.in Delivered-To: x14307373@homiemail-mx22.g.dreamhost.com Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by homiemail-mx22.g.dreamhost.com (Postfix) with ESMTPS id 869795080873 for ; Tue, 25 Mar 2014 03:28:34 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; q=dns; s=default; b=ypsKhf12z4Hq/vqrPY4NYpMUiuVJs 7Mm6n3XaEKXZHsJhlPB0/0eWdXxx1YDPp+zU9sEXTkziTKXM9t1TSKfwTkI2WunS rbdLHnFUvwYj/G/WOrlLq1YIy/miYlcs3deaSQ9MSGmIQtLbj1iyfthg3kg0nBFt H1LsCETo/1VSWI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; s=default; bh=d2XdLk5J+cPRrgS+E0+iHNTM/Lg=; b=fdK Alzj68GaQEQhbxVQnnwZHspGFHvVx578KqMcLq/C6osUhVcsbAL0ZDieoc70yo3r OuEZCVpahNVp5H/xWl7GE497URCNcL8q0X/T3gMYOqs8OR6Mf2yVV6ACoMmoWnvH rpnKXUWipMFKu5bVkh+bAyP0w2lWlJK5N8upT4Ck= Received: (qmail 22285 invoked by alias); 25 Mar 2014 10:28:31 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 22272 invoked by uid 89); 25 Mar 2014 10:28:30 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.9 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.2 X-HELO: mx1.redhat.com Date: Tue, 25 Mar 2014 15:58:59 +0530 From: Siddhesh Poyarekar To: libc-alpha@sourceware.org Subject: [PATCH 1/4] Make bench.out in json format Message-ID: <20140325102859.GA1850@spoyarek.pnq.redhat.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.22.1-rc1 (2013-10-16) X-DH-Original-To: glibc@patchwork.siddhesh.in Hi, This patch changes the output format of the main benchmark output file (bench.out) to an extensible format. I chose JSON over XML because in addition to being extensible, it is also not too verbose. Additionally it has good support in python. The significant change I have made in terms of functionality is to put timing information as an attribute in JSON instead of a string and to do that, there is a separate program that prints out a JSON snippet mentioning the type of timing (hp_timing or clock_gettime). The mean timing has now changed from iterations per unit to actual timing per iteration. Siddhesh * benchtests/Makefile (timing-type): New binary. (bench-clean): Also remove bench-timing-type. (bench): New target for timing-type. (bench-func): Print output in JSON format. * benchtests/bench-skeleton.c (main): Print output in JSON format. * benchtests/bench-timing-type.c: New file. * benchtests/bench-timing.h (TIMING_TYPE): New macro. (TIMING_PRINT_STATS): Remove. * benchtests/scripts/bench.py (_print_arg_data): Store variant name separately. --- benchtests/Makefile | 23 +++++++++++++++++++---- benchtests/bench-skeleton.c | 18 +++++++++++++++--- benchtests/bench-timing-type.c | 27 +++++++++++++++++++++++++++ benchtests/bench-timing.h | 14 ++++---------- benchtests/scripts/bench.py | 2 +- 5 files changed, 66 insertions(+), 18 deletions(-) create mode 100644 benchtests/bench-timing-type.c diff --git a/benchtests/Makefile b/benchtests/Makefile index b331d1a..be11708 100644 --- a/benchtests/Makefile +++ b/benchtests/Makefile @@ -98,11 +98,14 @@ run-bench = $(test-wrapper-env) \ GCONV_PATH=$(common-objpfx)iconvdata LC_ALL=C \ $($*-ENV) $(rtld-prefix) $${run} +timing-type := $(objpfx)bench-timing-type + bench-clean: rm -f $(binaries-bench) $(addsuffix .o,$(binaries-bench)) rm -f $(binaries-benchset) $(addsuffix .o,$(binaries-benchset)) + rm -f $(timing-type) $(addsuffix .o,$(timing-type)) -bench: bench-set bench-func +bench: $(timing-type) bench-set bench-func bench-set: $(binaries-benchset) for run in $^; do \ @@ -110,17 +113,29 @@ bench-set: $(binaries-benchset) $(run-bench) > $${run}.out; \ done +# Build and execute the benchmark functions. This target generates JSON +# formatted bench.out. Each of the programs produce independent JSON output, +# so one could even execute them individually and process it using any JSON +# capable language or tool. bench-func: $(binaries-bench) - { for run in $^; do \ + { echo "{"; \ + $(timing-type); \ + echo " ,\"functions\": {"; \ + for run in $^; do \ + if ! [ "x$${run}" = "x$<" ]; then \ + echo ","; \ + fi; \ echo "Running $${run}" >&2; \ $(run-bench); \ - done; } > $(objpfx)bench.out-tmp; \ + done; \ + echo " }"; \ + echo "}"; } > $(objpfx)bench.out-tmp; \ if [ -f $(objpfx)bench.out ]; then \ mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \ fi; \ mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out -$(binaries-bench) $(binaries-benchset): %: %.o \ +$(timing-type) $(binaries-bench) $(binaries-benchset): %: %.o \ $(sort $(filter $(common-objpfx)lib%,$(link-libc))) \ $(addprefix $(csu-objpfx),start.o) $(+preinit) $(+postinit) $(+link) diff --git a/benchtests/bench-skeleton.c b/benchtests/bench-skeleton.c index 4290e76..faef7eb 100644 --- a/benchtests/bench-skeleton.c +++ b/benchtests/bench-skeleton.c @@ -59,8 +59,13 @@ main (int argc, char **argv) iters = 1000 * res; + /* Begin function. */ + printf ("\"%s\": {\n", FUNCNAME); + for (int v = 0; v < NUM_VARIANTS; v++) { + if (v) + putc (',', stdout); /* Run for approximately DURATION seconds. */ clock_gettime (CLOCK_MONOTONIC_RAW, &runtime); runtime.tv_sec += DURATION; @@ -86,7 +91,6 @@ main (int argc, char **argv) min = cur; TIMING_ACCUM (total, cur); - d_total_i += iters; } struct timespec curtime; @@ -104,9 +108,17 @@ main (int argc, char **argv) d_total_s = total; d_iters = iters; - TIMING_PRINT_STATS (VARIANT (v), d_total_s, d_iters, d_total_i, max, - min); + printf ("\"%s\": {\n", VARIANT (v)); + printf ("\"duration\": %g, \"iterations\": %g, " + "\"max\": %g, \"min\": %g, \"mean\": %g\n", + d_total_s, d_total_i, max / d_iters, min / d_iters, + d_total_s / d_total_i); + + puts ("}"); } + /* End function. */ + puts ("}"); + return 0; } diff --git a/benchtests/bench-timing-type.c b/benchtests/bench-timing-type.c new file mode 100644 index 0000000..903a61f --- /dev/null +++ b/benchtests/bench-timing-type.c @@ -0,0 +1,27 @@ +/* Print out the timing type used by the benchmark run. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "bench-timing.h" +#include + +int +main (int argc, char **argv) +{ + printf ("\"timing-type\": \"%s\"\n", TIMING_TYPE); + return 0; +} diff --git a/benchtests/bench-timing.h b/benchtests/bench-timing.h index 13fc946..ccde601 100644 --- a/benchtests/bench-timing.h +++ b/benchtests/bench-timing.h @@ -25,6 +25,8 @@ hp_timing_t _dl_hp_timing_overhead; typedef hp_timing_t timing_t; +# define TIMING_TYPE "hp_timing" + # define TIMING_INIT(res) \ ({ \ HP_TIMING_DIFF_INIT(); \ @@ -35,16 +37,13 @@ typedef hp_timing_t timing_t; # define TIMING_DIFF(diff, start, end) HP_TIMING_DIFF ((diff), (start), (end)) # define TIMING_ACCUM(sum, diff) HP_TIMING_ACCUM_NT ((sum), (diff)) -# define TIMING_PRINT_STATS(func, d_total_s, d_iters, d_total_i, max, min) \ - printf ("%s: ITERS:%g: TOTAL:%gMcy, MAX:%gcy, MIN:%gcy, %g calls/Mcy\n", \ - (func), (d_total_i), (d_total_s) * 1e-6, (max) / (d_iters), \ - (min) / (d_iters), 1e6 * (d_total_i) / (d_total_s)); - #else #include typedef uint64_t timing_t; +# define TIMING_TYPE "clock_gettime" + /* Measure the resolution of the clock so we can scale the number of benchmark iterations by this value. */ # define TIMING_INIT(res) \ @@ -64,11 +63,6 @@ typedef uint64_t timing_t; # define TIMING_DIFF(diff, start, end) (diff) = (end) - (start) # define TIMING_ACCUM(sum, diff) (sum) += (diff) -# define TIMING_PRINT_STATS(func, d_total_s, d_iters, d_total_i, max, min) \ - printf ("%s: ITERS:%g: TOTAL:%gs, MAX:%gns, MIN:%gns, %g iter/s\n", \ - (func), (d_total_i), (d_total_s) * 1e-9, (max) / (d_iters), \ - (min) / (d_iters), 1e9 * (d_total_i) / (d_total_s)) - #endif #define TIMING_PRINT_MEAN(d_total_s, d_iters) \ diff --git a/benchtests/scripts/bench.py b/benchtests/scripts/bench.py index e500a33..90317b5 100755 --- a/benchtests/scripts/bench.py +++ b/benchtests/scripts/bench.py @@ -172,7 +172,7 @@ def _print_arg_data(func, directives, all_vals): # Members for the variants structure list that we will # print later. - variants.append(' {"%s(%s)", %d, in%d},' % (func, k, len(vals), i)) + variants.append(' {"%s", %d, in%d},' % (k, len(vals), i)) print(ARGS_TEMPLATE % {'argnum': i, 'num_args': len(vals), 'args': '\n'.join(out)})