From patchwork Tue Mar 25 10:30:17 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siddhesh Poyarekar X-Patchwork-Id: 266 Return-Path: X-Original-To: siddhesh@wilcox.dreamhost.com Delivered-To: siddhesh@wilcox.dreamhost.com Received: from homiemail-mx21.g.dreamhost.com (caibbdcaaahb.dreamhost.com [208.113.200.71]) by wilcox.dreamhost.com (Postfix) with ESMTP id B2B47360115 for ; Tue, 25 Mar 2014 03:29:47 -0700 (PDT) Received: by homiemail-mx21.g.dreamhost.com (Postfix, from userid 14307373) id 5FCAA1F4B905; Tue, 25 Mar 2014 03:29:47 -0700 (PDT) X-Original-To: glibc@patchwork.siddhesh.in Delivered-To: x14307373@homiemail-mx21.g.dreamhost.com Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by homiemail-mx21.g.dreamhost.com (Postfix) with ESMTPS id 3A468D9B96E for ; Tue, 25 Mar 2014 03:29:47 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; q=dns; s=default; b=HXtlYYzsFCU6Ix0LQLnf3C6WGHwnG 8GGhJWiwvrlyehV9oGcWFRiIuegqIZdrPCOjKxTOwZurwXNBVBbHHBigT4zZB9LT WI69wMKaJfhw1wbGYoZ6AX5mnC+sZW5G97dxAwbZ9yOkRzHgkAfdkL/rrBtw/Cou SjvOAqT3224Nbs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; s=default; bh=sb6rfq0tmE3qVvdnHOqnZpeccUE=; b=lBH BLVXcNLTuEKFq2gkt95Y2xNgYS59st5ny7F7WG4dfX7/NeCwQIJ27rspk5Zvu2o8 ULfw+11/E8HLlKkNVu4At7qXNHGibYhFMX/lRmGvW3WYVIm+b1EEs/74Z23hPpA7 lVYkjxAKR7UbDZZ4W/uo+HLHDu0/3rvz/QKBVoCI= Received: (qmail 25260 invoked by alias); 25 Mar 2014 10:29:43 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 25237 invoked by uid 89); 25 Mar 2014 10:29:42 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.9 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.2 X-HELO: mx1.redhat.com Date: Tue, 25 Mar 2014 16:00:17 +0530 From: Siddhesh Poyarekar To: libc-alpha@sourceware.org Subject: [PATCH 4/4] Validate bench.out against a JSON schema Message-ID: <20140325103017.GD1850@spoyarek.pnq.redhat.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.22.1-rc1 (2013-10-16) X-DH-Original-To: glibc@patchwork.siddhesh.in Hi, The first patch in this series converted the benchmark output into JSON format. This patch adds a JSON schema that describes the format of the benchmark output. There is also a script that validates the generated bench.out against the schema at the end of `make bench`. Siddhesh * benchtests/scripts/benchout.schema.json: New file. * benchtests/scripts/validate_benchout.py: New script. * benchtests/Makefile (bench-func): Call it. --- benchtests/Makefile | 2 + benchtests/scripts/benchout.schema.json | 57 ++++++++++++++++++++++++ benchtests/scripts/validate_benchout.py | 78 +++++++++++++++++++++++++++++++++ 3 files changed, 137 insertions(+) create mode 100644 benchtests/scripts/benchout.schema.json create mode 100755 benchtests/scripts/validate_benchout.py diff --git a/benchtests/Makefile b/benchtests/Makefile index 1669b6f..4b5b568 100644 --- a/benchtests/Makefile +++ b/benchtests/Makefile @@ -143,6 +143,8 @@ bench-func: $(binaries-bench) mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \ fi; \ mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out + $(.)scripts/validate_benchout.py $(objpfx)bench.out \ + $(.)scripts/benchout.schema.json $(timing-type) $(binaries-bench) $(binaries-benchset): %: %.o \ $(sort $(filter $(common-objpfx)lib%,$(link-libc))) \ diff --git a/benchtests/scripts/benchout.schema.json b/benchtests/scripts/benchout.schema.json new file mode 100644 index 0000000..3f39823 --- /dev/null +++ b/benchtests/scripts/benchout.schema.json @@ -0,0 +1,57 @@ +{ + "title": "benchmark", + "type": "object", + "properties": { + "machine": { + "type": "object", + "properties": { + "vendor-id": {"type": "string"}, + "cpu-family": {"type": "number"}, + "model": {"type": "number"}, + "cpu-cores": {"type": "number"}, + "processors": {"type": "number"}, + "cache-size": {"type": "number"}, + "memory": {"type": "number"}, + "swap": {"type": "number"} + }, + "required": ["vendor-id", "cpu-family", "model", "cpu-cores", + "processors", "cache-size", "memory", "swap"] + }, + "timing-type": { + "type": "string" + }, + "functions": { + "title": "Associative array of functions", + "type": "object", + "patternProperties": { + "^[_a-zA-Z][_a-zA-Z0-9]+$": { + "title": "Function names", + "type": "object", + "patternProperties": { + "^[_a-zA-Z0-9]*$": { + "title": "Function variants", + "type": "object", + "properties": { + "duration": {"type": "number"}, + "iterations": {"type": "number"}, + "max": {"type": "number"}, + "min": {"type": "number"}, + "mean": {"type": "number"}, + "timings": { + "type": "array", + "items": {"type": "number"} + } + }, + "required": ["duration", "iterations", "max", "min", "mean"], + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "minProperties": 1 + } + }, + "required": ["machine", "timing-type", "functions"], + "additionalProperties": false +} diff --git a/benchtests/scripts/validate_benchout.py b/benchtests/scripts/validate_benchout.py new file mode 100755 index 0000000..b6a34e2 --- /dev/null +++ b/benchtests/scripts/validate_benchout.py @@ -0,0 +1,78 @@ +#!/usr/bin/python +# Copyright (C) 2014 Free Software Foundation, Inc. +# This file is part of the GNU C Library. +# +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . +"""Benchmark output validator + +Given a benchmark output file in json format and a benchmark schema file, +validate the output against the schema. +""" + +from __future__ import print_function +import jsonschema +import json +import sys +import os + + +def validate_bench(benchfile, schemafile): + """Validate benchmark file + + Validate a benchmark output file against a JSON schema. + + Args: + benchfile: The file name of the bench.out file. + schemafile: The file name of the JSON schema file to validate + bench.out against. + + Exceptions: + jsonschema.ValidationError: When bench.out is not valid + jsonschema.SchemaError: When the JSON schema is not valid + IOError: If any of the files are not found. + """ + with open(benchfile, 'r') as bfile: + with open(schemafile, 'r') as sfile: + bench = json.load(bfile) + schema = json.load(sfile) + jsonschema.validate(bench, schema) + + # If we reach here, we're all good. + print("Benchmark output in %s is valid." % benchfile) + + +def main(args): + """Main entry point + + Args: + args: The command line arguments to the program + + Returns: + 0 on success or a non-zero failure code + + Exceptions: + Exceptions thrown by validate_bench + """ + if len(args) != 2: + print("Usage: %s " % sys.argv[0], + file=sys.stderr) + return os.EX_USAGE + + validate_bench(args[0], args[1]) + return os.EX_OK + + +if __name__ == '__main__': + sys.exit(main(sys.argv[1:]))