Add support for multiple data points to perf tests
Commit Message
Hi.
This patch does two things.
1) Add support for multiple data points.
2) Move the "report" output from perftest.log to perftest.sum.
I want to record the raw data somewhere, and a bit of statistical analysis
(standard deviation left for another day), but I also don't want
it to clutter up the basic report.
This patch takes a cue from gdb.{sum,log} and does the same thing
with perftest.{sum,log}.
Regression tested on amd64-linux.
2015-07-26 Doug Evans <xdje42@gmail.com>
* lib/perftest/reporter.py (SUM_FILE_NAME): New global.
(LOG_FILE_NAME): New global.
(TextReporter.__init__): Initialize self.txt_sum.
(TextReporter.report): Add support for multiple data-points.
Move report to perftest.sum, put raw data in perftest.log.
(TextReporter.start): Open sum and log files.
(TextReporter.end): Close sum and log files.
* lib/perftest/testresult.py (SingleStatisticTestResult.record): Handle
multiple data-points.
Comments
Doug Evans <xdje42@gmail.com> writes:
> This patch does two things.
> 1) Add support for multiple data points.
> 2) Move the "report" output from perftest.log to perftest.sum.
>
What does perftest.sum look like if we write test results by a different
reporter? say xml reporter or jason reporter?
> I want to record the raw data somewhere, and a bit of statistical analysis
> (standard deviation left for another day), but I also don't want
> it to clutter up the basic report.
> This patch takes a cue from gdb.{sum,log} and does the same thing
> with perftest.{sum,log}.
I'd like pertest framework only writes raw data down in some formats
(such as text, xml or jason) without any analysis. Statistical analysis
can be performed in a post-mortem way by other scripts.
On Mon, Jul 27, 2015 at 7:20 AM, Yao Qi <qiyaoltc@gmail.com> wrote:
> Doug Evans <xdje42@gmail.com> writes:
>
>> This patch does two things.
>> 1) Add support for multiple data points.
>> 2) Move the "report" output from perftest.log to perftest.sum.
>>
>
> What does perftest.sum look like if we write test results by a different
> reporter? say xml reporter or jason reporter?
Hi.
This question feels misguided, perhaps I haven't
sufficiently explained what I'm doing.
These same questions can be asked of perftest.log in the current tree.
All I've done is rename perftest.log to perftest.sum,
and then put more test results in text form in perftest.log.
I *could* put it all in one file, but I like what's in perftest.log today,
(though there are a few tweaks I'd like to make),
I'm just moving it to perftest.sum.
Following the pattern of file naming already in use
will be helpful to everyone using the perf testsuite.
If/when we have other forms of output I would expect them
to not go to .log or .sum files.
>> I want to record the raw data somewhere, and a bit of statistical analysis
>> (standard deviation left for another day), but I also don't want
>> it to clutter up the basic report.
>> This patch takes a cue from gdb.{sum,log} and does the same thing
>> with perftest.{sum,log}.
>
> I'd like pertest framework only writes raw data down in some formats
> (such as text, xml or jason) without any analysis. Statistical analysis
> can be performed in a post-mortem way by other scripts.
We can generate a .csv file with the raw data if/when the need arises.
[or whatever is easiest for post-processing scripts to read]
A human readable form with basic statistical analysis
is useful today, and doable in 5 lines of code.
Let's add the complexity when there's a demonstrated
need for it.
Doug Evans <xdje42@gmail.com> writes:
> All I've done is rename perftest.log to perftest.sum,
> and then put more test results in text form in perftest.log.
Oh, I see.
> I *could* put it all in one file, but I like what's in perftest.log today,
> (though there are a few tweaks I'd like to make),
> I'm just moving it to perftest.sum.
> Following the pattern of file naming already in use
> will be helpful to everyone using the perf testsuite.
Right. To be clear, with your patch applied, raw data is saved in
perftest.sum and some basic statistical analysis is output to
perftest.log.
>
> If/when we have other forms of output I would expect them
> to not go to .log or .sum files.
>
I agree.
>>> I want to record the raw data somewhere, and a bit of statistical analysis
>>> (standard deviation left for another day), but I also don't want
>>> it to clutter up the basic report.
>>> This patch takes a cue from gdb.{sum,log} and does the same thing
>>> with perftest.{sum,log}.
>>
>> I'd like pertest framework only writes raw data down in some formats
>> (such as text, xml or jason) without any analysis. Statistical analysis
>> can be performed in a post-mortem way by other scripts.
>
> We can generate a .csv file with the raw data if/when the need arises.
> [or whatever is easiest for post-processing scripts to read]
>
> A human readable form with basic statistical analysis
> is useful today, and doable in 5 lines of code.
> Let's add the complexity when there's a demonstrated
> need for it.
OK, no problem. I don't have other questions, and your patch looks good
to me.
On Wed, Jul 29, 2015 at 1:45 AM, Yao Qi <qiyaoltc@gmail.com> wrote:
> Doug Evans <xdje42@gmail.com> writes:
>
>> All I've done is rename perftest.log to perftest.sum,
>> and then put more test results in text form in perftest.log.
>
> Oh, I see.
>
>> I *could* put it all in one file, but I like what's in perftest.log today,
>> (though there are a few tweaks I'd like to make),
>> I'm just moving it to perftest.sum.
>> Following the pattern of file naming already in use
>> will be helpful to everyone using the perf testsuite.
>
> Right. To be clear, with your patch applied, raw data is saved in
> perftest.sum and some basic statistical analysis is output to
> perftest.log.
>
>>
>> If/when we have other forms of output I would expect them
>> to not go to .log or .sum files.
>>
>
> I agree.
>
>>>> I want to record the raw data somewhere, and a bit of statistical analysis
>>>> (standard deviation left for another day), but I also don't want
>>>> it to clutter up the basic report.
>>>> This patch takes a cue from gdb.{sum,log} and does the same thing
>>>> with perftest.{sum,log}.
>>>
>>> I'd like pertest framework only writes raw data down in some formats
>>> (such as text, xml or jason) without any analysis. Statistical analysis
>>> can be performed in a post-mortem way by other scripts.
>>
>> We can generate a .csv file with the raw data if/when the need arises.
>> [or whatever is easiest for post-processing scripts to read]
>>
>> A human readable form with basic statistical analysis
>> is useful today, and doable in 5 lines of code.
>> Let's add the complexity when there's a demonstrated
>> need for it.
>
> OK, no problem. I don't have other questions, and your patch looks good
> to me.
Thanks. Committed.
@@ -13,6 +13,15 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
+# Text reports are written here.
+# This is the perftest counterpart to gdb.sum.
+SUM_FILE_NAME = "perftest.sum"
+
+# Raw data that went into the report is written here.
+# This is the perftest counterpart to gdb.log.
+LOG_FILE_NAME = "perftest.log"
+
+
class Reporter(object):
"""Base class of reporter to report test results in a certain format.
@@ -43,22 +52,34 @@ class Reporter(object):
"""
raise NotImplementedError("Abstract Method:end.")
+
class TextReporter(Reporter):
"""Report results in a plain text file 'perftest.log'."""
def __init__(self, append):
super (TextReporter, self).__init__(Reporter(append))
+ self.txt_sum = None
self.txt_log = None
- def report(self, *args):
- self.txt_log.write(' '.join(str(arg) for arg in args))
- self.txt_log.write('\n')
+ def report(self, test_name, measurement_name, data_points):
+ if len(data_points) == 0:
+ self.txt_sum.write("%s %s *no data recorded*\n" % (
+ test_name, measurement_name))
+ return
+ average = sum(data_points) / len(data_points)
+ data_min = min(data_points)
+ data_max = max(data_points)
+ self.txt_sum.write("%s %s %s\n" % (
+ test_name, measurement_name, average))
+ self.txt_log.write("%s %s %s, min %s, max %s, data %s\n" % (
+ test_name, measurement_name, average, data_min, data_max,
+ data_points))
def start(self):
- if self.append:
- self.txt_log = open ("perftest.log", 'a+');
- else:
- self.txt_log = open ("perftest.log", 'w');
+ mode = "a+" if self.append else "w"
+ self.txt_sum = open (SUM_FILE_NAME, mode);
+ self.txt_log = open (LOG_FILE_NAME, mode);
def end(self):
+ self.txt_sum.close ()
self.txt_log.close ()
@@ -35,7 +35,10 @@ class SingleStatisticTestResult(TestResult):
self.results = dict ()
def record(self, parameter, result):
- self.results[parameter] = result
+ if parameter in self.results:
+ self.results[parameter].append(result)
+ else:
+ self.results[parameter] = [result]
def report(self, reporter, name):
reporter.start()