From patchwork Thu Dec 9 03:55:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Frank Ch. Eigler" X-Patchwork-Id: 48656 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E31FE3858435 for ; Thu, 9 Dec 2021 03:55:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E31FE3858435 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1639022154; bh=V62fdoDzugL1IXu7/LaZYTfitN8d7zJ3ez0DxWz7fBU=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Help: List-Subscribe:From:Reply-To:From; b=f3Gknr/3fUN2WHlH60++nvHM5VygV2qsLHW/AmtDbbJxxvOuF7UBbWHuoFjeFaSog TtdvOB2ECJpU9HieNXG4jWtqYXPWheRq4+JQ/ofC1SRzq417uQxHeDCqrcg8b07gGE +wewygJHQKfgNiRhsWNZv7Hz3e44qbGfuCwrqInk= X-Original-To: elfutils-devel@sourceware.org Delivered-To: elfutils-devel@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 8D7C23858D28 for ; Thu, 9 Dec 2021 03:55:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8D7C23858D28 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-138-UIMhM26EMLW614aPJ3dzVQ-1; Wed, 08 Dec 2021 22:55:44 -0500 X-MC-Unique: UIMhM26EMLW614aPJ3dzVQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4A4EA8015CD for ; Thu, 9 Dec 2021 03:55:43 +0000 (UTC) Received: from redhat.com (unknown [10.2.16.5]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 19AC55D6CF for ; Thu, 9 Dec 2021 03:55:43 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1mvAWz-0005Yo-KS for elfutils-devel@sourceware.org; Wed, 08 Dec 2021 22:55:42 -0500 Date: Wed, 8 Dec 2021 22:55:41 -0500 To: elfutils-devel@sourceware.org Subject: patch rfc: PR28661: debuginfod thread-pool Message-ID: <20211209035541.GA21366@redhat.com> MIME-Version: 1.0 User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-Patchwork-Original-From: "Frank Ch. Eigler via Elfutils-devel" From: "Frank Ch. Eigler" Reply-To: "Frank Ch. Eigler" Errors-To: elfutils-devel-bounces+patchwork=sourceware.org@sourceware.org Sender: "Elfutils-devel" Hi - While I think this patch itself is fine, and works around the libmicrohttpd bug that motivated it, I don't know how to test it seriously in the testsuite. (We can certainly try few -C options for parsing & operability.) The error edge cases only appear to occur under very high load and task limits such as those imposed by systemd cgroups. Author: Frank Ch. Eigler Date: Wed Dec 8 22:41:47 2021 -0500 PR28661: debuginfo connection thread pool support Add an option -C, which activates libmicrohttpd's thread-pool mode for handling incoming http connections. Add libmicrohttpd error-logging callback function so as to receive indication of its internal errors, and relay counts to our metrics. Some of these internal errors tipped us off to a microhttpd bug that thread pooling works around. Document in debuginfod.8 page. Hand-tested against "ulimit -u NNN" shells. Signed-off-by: Frank Ch. Eigler diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog index 21d0721ef080..125e7d816f51 100644 --- a/debuginfod/ChangeLog +++ b/debuginfod/ChangeLog @@ -1,3 +1,11 @@ +2021-12-08 Frank Ch. Eigler + + * debuginfod.cxx (connection_pool): New global. + (parse_opt): Parse & check -C option to set it. + (error_cb): New callback for libmicrohttpd error counting. + (main): Activate MHD_OPTION_THREAD_POOL_SIZE if appropriate. + Activate error_cb. + 2021-12-01 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Free tmp_url on diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 0d3f02978ee2..bb0c6cd65670 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -352,7 +352,9 @@ static const struct argp_option options[] = { "rescan-time", 't', "SECONDS", 0, "Number of seconds to wait between rescans, 0=disable.", 0 }, { "groom-time", 'g', "SECONDS", 0, "Number of seconds to wait between database grooming, 0=disable.", 0 }, { "maxigroom", 'G', NULL, 0, "Run a complete database groom/shrink pass at startup.", 0 }, - { "concurrency", 'c', "NUM", 0, "Limit scanning thread concurrency to NUM.", 0 }, + { "concurrency", 'c', "NUM", 0, "Limit scanning thread concurrency to NUM, default=#CPUs.", 0 }, + { "connection-pool", 'C', "NUM", OPTION_ARG_OPTIONAL, + "Use webapi connection pool with NUM threads, default=unlim.", 0 }, { "include", 'I', "REGEX", 0, "Include files matching REGEX, default=all.", 0 }, { "exclude", 'X', "REGEX", 0, "Exclude files matching REGEX, default=none.", 0 }, { "port", 'p', "NUM", 0, "HTTP port to listen on, default 8002.", 0 }, @@ -411,6 +413,7 @@ static unsigned rescan_s = 300; static unsigned groom_s = 86400; static bool maxigroom = false; static unsigned concurrency = std::thread::hardware_concurrency() ?: 1; +static int connection_pool = 0; static set source_paths; static bool scan_files = false; static map scan_archives; @@ -557,6 +560,16 @@ parse_opt (int key, char *arg, concurrency = (unsigned) atoi(arg); if (concurrency < 1) concurrency = 1; break; + case 'C': + if (arg) + { + connection_pool = atoi(arg); + if (connection_pool < 2) + argp_failure(state, 1, EINVAL, "-C NUM minimum 2"); + } + else // arg not given + connection_pool = std::thread::hardware_concurrency() * 2 ?: 2; + break; case 'I': // NB: no problem with unconditional free here - an earlier failed regcomp would exit program if (passive_p) @@ -3684,7 +3698,13 @@ sigusr2_handler (int /* sig */) } - +static void // error logging callback from libmicrohttpd internals +error_cb (void *arg, const char *fmt, va_list ap) +{ + (void) arg; + inc_metric("error_count","libmicrohttpd",fmt); + (void) vdprintf (STDERR_FILENO, fmt, ap); +} // A user-defined sqlite function, to score the sharedness of the @@ -3829,7 +3849,7 @@ main (int argc, char *argv[]) // Start httpd server threads. Separate pool for IPv4 and IPv6, in // case the host only has one protocol stack. - MHD_Daemon *d4 = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION + MHD_Daemon *d4 = MHD_start_daemon ((connection_pool ? 0 : MHD_USE_THREAD_PER_CONNECTION) #if MHD_VERSION >= 0x00095300 | MHD_USE_INTERNAL_POLLING_THREAD #else @@ -3839,8 +3859,11 @@ main (int argc, char *argv[]) http_port, NULL, NULL, /* default accept policy */ handler_cb, NULL, /* handler callback */ + MHD_OPTION_EXTERNAL_LOGGER, error_cb, NULL, + (connection_pool ? MHD_OPTION_THREAD_POOL_SIZE : MHD_OPTION_END), + (connection_pool ? (int)connection_pool : MHD_OPTION_END), MHD_OPTION_END); - MHD_Daemon *d6 = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION + MHD_Daemon *d6 = MHD_start_daemon ((connection_pool ? 0 : MHD_USE_THREAD_PER_CONNECTION) #if MHD_VERSION >= 0x00095300 | MHD_USE_INTERNAL_POLLING_THREAD #else @@ -3851,6 +3874,9 @@ main (int argc, char *argv[]) http_port, NULL, NULL, /* default accept policy */ handler_cb, NULL, /* handler callback */ + MHD_OPTION_EXTERNAL_LOGGER, error_cb, NULL, + (connection_pool ? MHD_OPTION_THREAD_POOL_SIZE : MHD_OPTION_END), + (connection_pool ? (int)connection_pool : MHD_OPTION_END), MHD_OPTION_END); if (d4 == NULL && d6 == NULL) // neither ipv4 nor ipv6? boo @@ -3904,6 +3930,8 @@ main (int argc, char *argv[]) if (! passive_p) obatched(clog) << "search concurrency " << concurrency << endl; + obatched(clog) << "webapi connection pool " << connection_pool + << (connection_pool ? "" : " (unlimited)") << endl; if (! passive_p) obatched(clog) << "rescan time " << rescan_s << endl; obatched(clog) << "fdcache fds " << fdcache_fds << endl; diff --git a/doc/ChangeLog b/doc/ChangeLog index 7a73fa107bf3..f25bcd2ee2b3 100644 --- a/doc/ChangeLog +++ b/doc/ChangeLog @@ -1,3 +1,8 @@ +2021-12-08 Frank Ch. Eigler + + PR28661 + * debuginfod.8 (-C): Document new flag. + 2021-11-05 Frank Ch. Eigler PR28430 diff --git a/doc/debuginfod.8 b/doc/debuginfod.8 index 1e56f6568322..ee8e4078e5b5 100644 --- a/doc/debuginfod.8 +++ b/doc/debuginfod.8 @@ -205,6 +205,23 @@ This important for controlling CPU-intensive operations like parsing an ELF file and especially decompressing archives. The default is the number of processors on the system; the minimum is 1. +.TP +.B "\-C" "\-C=NUM" "\-\-connection\-pool" "\-\-connection\-pool=NUM" +Set the size of the pool of threads serving webapi queries. The +following table summarizes the interpretaton of this option and its +optional NUM parameter. +.TS +l l. +no option clone new thread for every request, no fixed pool +\-C use a fixed thread pool sized automatically +\-C=NUM use a fixed thread pool sized NUM, minimum 2 +.TE + +The first mode is useful for friendly bursty traffic. The second mode +is a simple and safe configuration based on the number of processors. +The third mode is suitable for tuned load-limiting configurations +facing unruly traffic. + .TP .B "\-L" Traverse symbolic links encountered during traversal of the PATHs,