Bump up the runtime for "short" benchmarks

Message ID 20190626153841.GA7331@bell-sw.com
State Committed
Headers

Commit Message

Anton Youdkevitch June 26, 2019, 3:38 p.m. UTC
  Some benchmarks with a very short runtime shows significantly
different results across runs on Aarch64 - up to tens of percents.
Increasing the runtime to 100ms+ makes the deviation under 5%.

Tested on Aarch64 and x86-64.

	* benchtests/bench-memccpy.c: replace INNER_LOOP_ITERS
	with INNER_LOOP_ITERS_LARGE
	* benchtests/bench-memchr.c: Likewise
	* benchtests/bench-rawmemchr.c: Likewise
	* benchtests/bench-strcat.c: Likewise
	* benchtests/bench-strchr.c: Likewise
	* benchtests/bench-string.h: Likewise
	* benchtests/bench-strlen.c: Likewise
	* benchtests/bench-strncpy.c: Likewise
	* benchtests/bench-strnlen.c: Likewise

Looks OK?
  

Comments

Carlos O'Donell June 26, 2019, 9:32 p.m. UTC | #1
On 6/26/19 11:38 AM, Anton Youdkevitch wrote:
> Some benchmarks with a very short runtime shows significantly
> different results across runs on Aarch64 - up to tens of percents.
> Increasing the runtime to 100ms+ makes the deviation under 5%.
> 
> Tested on Aarch64 and x86-64.
> 
> 	* benchtests/bench-memccpy.c: replace INNER_LOOP_ITERS
> 	with INNER_LOOP_ITERS_LARGE
> 	* benchtests/bench-memchr.c: Likewise
> 	* benchtests/bench-rawmemchr.c: Likewise
> 	* benchtests/bench-strcat.c: Likewise
> 	* benchtests/bench-strchr.c: Likewise
> 	* benchtests/bench-string.h: Likewise
> 	* benchtests/bench-strlen.c: Likewise
> 	* benchtests/bench-strncpy.c: Likewise
> 	* benchtests/bench-strnlen.c: Likewise
> 
> Looks OK?

Do you have commit or copyright assignment?

OK for master.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
  
Anton Youdkevitch June 27, 2019, 9:33 a.m. UTC | #2
On 6/26/2019 11:32 PM, Carlos O'Donell wrote:
> On 6/26/19 11:38 AM, Anton Youdkevitch wrote:
>> Some benchmarks with a very short runtime shows significantly
>> different results across runs on Aarch64 - up to tens of percents.
>> Increasing the runtime to 100ms+ makes the deviation under 5%.
>>
>> Tested on Aarch64 and x86-64.
>>
>> 	* benchtests/bench-memccpy.c: replace INNER_LOOP_ITERS
>> 	with INNER_LOOP_ITERS_LARGE
>> 	* benchtests/bench-memchr.c: Likewise
>> 	* benchtests/bench-rawmemchr.c: Likewise
>> 	* benchtests/bench-strcat.c: Likewise
>> 	* benchtests/bench-strchr.c: Likewise
>> 	* benchtests/bench-string.h: Likewise
>> 	* benchtests/bench-strlen.c: Likewise
>> 	* benchtests/bench-strncpy.c: Likewise
>> 	* benchtests/bench-strnlen.c: Likewise
>>
>> Looks OK?
> Do you have commit or copyright assignment?
I don't. Previously my colleague that has did it on my behalf.

>
> OK for master.
>
> Reviewed-by: Carlos O'Donell <carlos@redhat.com>
>
  
Carlos O'Donell June 27, 2019, 4:56 p.m. UTC | #3
On 6/27/19 5:33 AM, Anton Youdkevitch wrote:
> On 6/26/2019 11:32 PM, Carlos O'Donell wrote:
>> On 6/26/19 11:38 AM, Anton Youdkevitch wrote:
>>> Some benchmarks with a very short runtime shows significantly
>>> different results across runs on Aarch64 - up to tens of percents.
>>> Increasing the runtime to 100ms+ makes the deviation under 5%.
>>>
>>> Tested on Aarch64 and x86-64.
>>>
>>>     * benchtests/bench-memccpy.c: replace INNER_LOOP_ITERS
>>>     with INNER_LOOP_ITERS_LARGE
>>>     * benchtests/bench-memchr.c: Likewise
>>>     * benchtests/bench-rawmemchr.c: Likewise
>>>     * benchtests/bench-strcat.c: Likewise
>>>     * benchtests/bench-strchr.c: Likewise
>>>     * benchtests/bench-string.h: Likewise
>>>     * benchtests/bench-strlen.c: Likewise
>>>     * benchtests/bench-strncpy.c: Likewise
>>>     * benchtests/bench-strnlen.c: Likewise
>>>
>>> Looks OK?
>> Do you have commit or copyright assignment?
> I don't. Previously my colleague that has did it on my behalf.

The modern FSF assignment process is fairly painless (all digital
depending on your country).

If you setup a "futures" assignment then we could just review and
take all future patches you post.

http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future

https://sourceware.org/glibc/wiki/Contribution%20checklist#FSF_copyright_Assignment
  

Patch

From 3e55575cde5905e6550bb33161dfda657283b823 Mon Sep 17 00:00:00 2001
From: Anton Youdkevitch <anton.youdkevitch@bell-sw.com>
Date: Wed, 26 Jun 2019 14:33:50 +0000
Subject: [PATCH] Bump up the runtime for "short" benchmarks

The runtime of the benchmarks that ran less than 100ms
on Aarch64 was increased. The new INNER_LOOP_ITERS_LARGE
constant for such benchmaks was introduced.
The goal for the deviation being under 5% between runs
is achieved.
---
 benchtests/bench-memccpy.c   | 2 +-
 benchtests/bench-memchr.c    | 2 +-
 benchtests/bench-rawmemchr.c | 2 +-
 benchtests/bench-strcat.c    | 2 +-
 benchtests/bench-strchr.c    | 2 +-
 benchtests/bench-string.h    | 1 +
 benchtests/bench-strlen.c    | 2 +-
 benchtests/bench-strncpy.c   | 2 +-
 benchtests/bench-strnlen.c   | 2 +-
 9 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/benchtests/bench-memccpy.c b/benchtests/bench-memccpy.c
index d3a2df5..46f96d5 100644
--- a/benchtests/bench-memccpy.c
+++ b/benchtests/bench-memccpy.c
@@ -41,7 +41,7 @@  static void
 do_one_test (impl_t *impl, void *dst, const void *src, int c, size_t len,
 	     size_t n)
 {
-  size_t i, iters = INNER_LOOP_ITERS;
+  size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   TIMING_NOW (start);
diff --git a/benchtests/bench-memchr.c b/benchtests/bench-memchr.c
index 9bd07eb..8a9909a 100644
--- a/benchtests/bench-memchr.c
+++ b/benchtests/bench-memchr.c
@@ -56,7 +56,7 @@  SIMPLE_MEMCHR (const CHAR *s, int c, size_t n)
 static void
 do_one_test (impl_t *impl, const CHAR *s, int c, size_t n)
 {
-  size_t i, iters = INNER_LOOP_ITERS;
+  size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   TIMING_NOW (start);
diff --git a/benchtests/bench-rawmemchr.c b/benchtests/bench-rawmemchr.c
index 44f3114..5cdfdf7 100644
--- a/benchtests/bench-rawmemchr.c
+++ b/benchtests/bench-rawmemchr.c
@@ -39,7 +39,7 @@  IMPL (generic_rawmemchr, 0)
 static void
 do_one_test (impl_t *impl, const char *s, int c, char *exp_res)
 {
-  size_t i, iters = INNER_LOOP_ITERS;
+  size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
   char *res = CALL (impl, s, c);
   if (res != exp_res)
diff --git a/benchtests/bench-strcat.c b/benchtests/bench-strcat.c
index 39d9fef..08841d0 100644
--- a/benchtests/bench-strcat.c
+++ b/benchtests/bench-strcat.c
@@ -51,7 +51,7 @@  IMPL (generic_strcat, 0)
 static void
 do_one_test (impl_t *impl, CHAR *dst, const CHAR *src)
 {
-  size_t k = STRLEN (dst), i, iters = INNER_LOOP_ITERS;
+  size_t k = STRLEN (dst), i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   if (CALL (impl, dst, src) != dst)
diff --git a/benchtests/bench-strchr.c b/benchtests/bench-strchr.c
index b8dba17..10d06fe 100644
--- a/benchtests/bench-strchr.c
+++ b/benchtests/bench-strchr.c
@@ -76,7 +76,7 @@  IMPL (STRCHR, 1)
 static void
 do_one_test (impl_t *impl, const CHAR *s, int c, const CHAR *exp_res)
 {
-  size_t i, iters = INNER_LOOP_ITERS;
+  size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   TIMING_NOW (start);
diff --git a/benchtests/bench-string.h b/benchtests/bench-string.h
index e7fb015..7d35791 100644
--- a/benchtests/bench-string.h
+++ b/benchtests/bench-string.h
@@ -125,6 +125,7 @@  extern impl_t __start_impls[], __stop_impls[];
 # define OPT_SEED 10002
 
 # define INNER_LOOP_ITERS 8192
+# define INNER_LOOP_ITERS_LARGE 131072
 # define INNER_LOOP_ITERS_MEDIUM 1024
 # define INNER_LOOP_ITERS_SMALL 32
 
diff --git a/benchtests/bench-strlen.c b/benchtests/bench-strlen.c
index 47363af..1997ad7 100644
--- a/benchtests/bench-strlen.c
+++ b/benchtests/bench-strlen.c
@@ -48,7 +48,7 @@  IMPL (STRLEN, 1)
 static void
 do_one_test (json_ctx_t *json_ctx, impl_t *impl, const CHAR *s, size_t exp_len)
 {
-  size_t len = CALL (impl, s), i, iters = INNER_LOOP_ITERS;
+  size_t len = CALL (impl, s), i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   if (len != exp_len)
diff --git a/benchtests/bench-strncpy.c b/benchtests/bench-strncpy.c
index 93bd6d5..c6a6799 100644
--- a/benchtests/bench-strncpy.c
+++ b/benchtests/bench-strncpy.c
@@ -54,7 +54,7 @@  typedef CHAR *(*proto_t) (CHAR *, const CHAR *, size_t);
 static void
 do_one_test (impl_t *impl, CHAR *dst, const CHAR *src, size_t len, size_t n)
 {
-  size_t i, iters = INNER_LOOP_ITERS;
+  size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   if (CALL (impl, dst, src, n) != STRNCPY_RESULT (dst, len, n))
diff --git a/benchtests/bench-strnlen.c b/benchtests/bench-strnlen.c
index e9e51b2..72ec303 100644
--- a/benchtests/bench-strnlen.c
+++ b/benchtests/bench-strnlen.c
@@ -51,7 +51,7 @@  IMPL (generic_strnlen, 0)
 static void
 do_one_test (impl_t *impl, const CHAR *s, size_t maxlen, size_t exp_len)
 {
-  size_t len = CALL (impl, s, maxlen), i, iters = INNER_LOOP_ITERS;
+  size_t len = CALL (impl, s, maxlen), i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
 
   if (len != exp_len)
-- 
2.7.4