From patchwork Wed May 25 18:44:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Sanci X-Patchwork-Id: 54385 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id AD3C8383066A for ; Wed, 25 May 2022 18:44:34 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org AD3C8383066A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1653504274; bh=/n+vCT7nWtIymO5yN2KsN+QbcisUr4TiyZRQVBNVrnE=; h=Date:Subject:To:List-Id:List-Unsubscribe:List-Archive:List-Help: List-Subscribe:From:Reply-To:From; b=F7v3TqnkyzGX1349l7qajybu3DDubPcA37nWzOFv+3xv5d4FaXsv0PveYv8czuaZL 6j05JMv33kiDDgvhgHDOqIuWUG4Cu34AyR98rA9cDrXYZzTLwop291M1nfIpazQq8P XHx6QH2A1BcsFSqnY5MJASTCcPFVC3XStYkJHh9E= X-Original-To: elfutils-devel@sourceware.org Delivered-To: elfutils-devel@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 2696C3856DC4 for ; Wed, 25 May 2022 18:44:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 2696C3856DC4 Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-263-ZWBed-aMPX6xhAR-4qZegA-1; Wed, 25 May 2022 14:44:22 -0400 X-MC-Unique: ZWBed-aMPX6xhAR-4qZegA-1 Received: by mail-io1-f71.google.com with SMTP id r10-20020a5edb4a000000b006656e9a2603so4714068iop.6 for ; Wed, 25 May 2022 11:44:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=/n+vCT7nWtIymO5yN2KsN+QbcisUr4TiyZRQVBNVrnE=; b=lzD0Rn3oLJQ/tx2tWh5+z53aiFeVgNgAYmeN3GuOaUik/F1V+b3Du4+QgeiWHNARZo e1Yk6yxxsmiiO8BJwR1BnqSAvwzonnJfYDWBCqON595H4rKTOmT7o9FThX8mwoPGb/B+ LU61FvyX+wEAJ7hDBxKRAUhw94ZyYwQ9aS32y7ToaTzRBeqSU/r8OONSy2LODkpqzNE3 kLP01oBbYKKkZaa92EbG4N0cgCJA8noE8TsMu3FSCuqFBtzj/FMHQ0lCQPJE69HJvSfi 6/7341+q2oT9RUJGRSte6W9efB4Fm5Piz0dpOFvKUQ8NuPiv1l+Nosk5kED5U+1Sty36 jvXQ== X-Gm-Message-State: AOAM53294gaP3u5QcUjqLmzLMb1YcxMSDUH9W59mAcCJsaqGaJQlukM1 V+Nh7XfdPjA1iykRdpUUqmaQmM92wCqtf0WAV9cvN//IhxSdfHbKR2Z8vrk+SlPEvhJAgjqhDAi 0Tnh82uvC6xmvnLd8btuYovAQRPqMlCjajHdb1t2p1A== X-Received: by 2002:a05:6e02:1284:b0:2d1:ec71:838e with SMTP id y4-20020a056e02128400b002d1ec71838emr231147ilq.183.1653504261808; Wed, 25 May 2022 11:44:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwTKdnO6va4J/MpJhYLvZb1O7P7X39mPwQrLH4pZtHIzt9l2YwERQ4BIQbKtYJezblM3Vqml1em6N3F6ICt+sw= X-Received: by 2002:a05:6e02:1284:b0:2d1:ec71:838e with SMTP id y4-20020a056e02128400b002d1ec71838emr231137ilq.183.1653504261579; Wed, 25 May 2022 11:44:21 -0700 (PDT) MIME-Version: 1.0 Date: Wed, 25 May 2022 14:44:10 -0400 Message-ID: Subject: [Bug debuginfod/28577] Make run-...-fd-prefetch-caches.sh test something To: elfutils-devel@sourceware.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_BADIPHTTP, KAM_LOTSOFHASH, NUMERIC_HTTP_ADDR, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Sanci via Elfutils-devel From: Noah Sanci Reply-To: Noah Sanci Errors-To: elfutils-devel-bounces+patchwork=sourceware.org@sourceware.org Sender: "Elfutils-devel" PR28577: Make run-debuginfod-fd-prefetch-caches.sh test something Update to the run-debuginfod-fd-prefetch to make the test more sound. From b84dd9b2b36812ea299ff189c7e5557863dd82d2 Mon Sep 17 00:00:00 2001 From: Noah Sanci Date: Fri, 13 May 2022 14:32:27 -0400 Subject: [PATCH] PR28577: Make run-debuginfod-fd-prefetch-caches.sh test something Update to the run-debuginfod-fd-prefetch to make the test more sound. --- tests/run-debuginfod-fd-prefetch-caches.sh | 116 +++++++++------------ 1 file changed, 48 insertions(+), 68 deletions(-) diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh index 66a7a083..50ffc4ac 100755 --- a/tests/run-debuginfod-fd-prefetch-caches.sh +++ b/tests/run-debuginfod-fd-prefetch-caches.sh @@ -21,22 +21,22 @@ # for test case debugging, uncomment: set -x unset VALGRIND_CMD +mkdir R Z -mkdir R -cp -rvp ${abs_srcdir}/debuginfod-rpms R -if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones - rm -vrf R/debuginfod-rpms/fedora31 -fi - -FDCACHE_MBS=$((1/1024/1024)) +FDCACHE_MBS=1 FDCACHE_FDS=1 -PREFETCH_MBS=$((1/1024/1024)) -PREFETCH_FDS=2 +PREFETCH_MBS=1 +PREFETCH_FDS=1 PREFETCH=1 # This variable is essential and ensures no time-race for claiming ports occurs # set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test base=8800 get_ports +cp -rvp ${abs_srcdir}/debuginfod-rpms R +if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones + rm -vrf R/debuginfod-rpms/fedora31 +fi +cp -rvp ${abs_srcdir}/debuginfod-tars Z DB=${PWD}/.debuginfod_tmp.sqlite tempfiles $DB @@ -47,88 +47,68 @@ export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache ###### rm -rf $DEBUGINFOD_CACHE_PATH rm -rf $DB + # Set mb values to ensure the caches certainly have enough space # To store the test files env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \ - --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \ - --fdcache-mbs=50 --fdcache-prefetch-mbs=50 \ - --fdcache-prefetch=$PREFETCH > vlog$PORT1 2>&1 & + --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS -vvvvv -g 0 -t 0 -R R \ + -Z .tar.bz2=bzcat Z --fdcache-mbs=50 --fdcache-prefetch-mbs=100 \ + --fdcache-prefetch=$PREFETCH -F > vlog$PORT1 2>&1 & PID1=$! tempfiles vlog$PORT1 errfiles vlog$PORT1 + # Server must become ready wait_ready $PORT1 'ready' 1 ######################################################################## -cp -rvp ${abs_srcdir}/debuginfod-rpms R -if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones - rm -vrf R/debuginfod-rpms/fedora31 -fi kill -USR1 $PID1 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 +wait_ready $PORT1 'thread_busy{role="scan"}' 0 # All rpms need to be in the index, except the dummy permission-000 one rpms=$(find R -name \*rpm | grep -v nothing | wc -l) wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs -# Wait till both files are in the index and scan/index fully finished +kill -USR1 $PID1 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 -export DEBUGINFOD_URLS="http://127.0.0.1:"$PORT1 +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 +wait_ready $PORT1 'thread_busy{role="scan"}' 0 -archive_hits=5 -SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1 -for i in $archive_hits -do - archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA -done +export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ +# load prefetch cache +testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb metrics=$(curl http://127.0.0.1:$PORT1/metrics) -regex="fdcache_op_count\{op=\"enqueue\"\} ([0-9]+).*fdcache_op_count\{op=\"evict\"\} ([0-9]+).*" -enqueue=0 -if [[ $metrics =~ $regex ]] -then - enqueue=${BASH_REMATCH[1]} - evict=${BASH_REMATCH[2]} +regex="fdcache_prefetch_count ([0-9])+" +# Check to see if prefetch cache is loaded +if [[ $metrics =~ $regex ]]; then + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then + err + fi else - err -fi -# This is an ad-hoc test than only works when FDCACHE_FDS < "archive_test"s above (5) -# otherwise there will be no eviction -if [[ $(( $enqueue-$FDCACHE_FDS )) -ne $evict ]] -then - err -fi -# Test prefetch cache -archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA + err +fi +# Ensure that searching for the source prefetched finds it in the prefetch cache +testrun ${abs_top_builddir}/debuginfod/debuginfod-find source cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c metrics=$(curl http://127.0.0.1:$PORT1/metrics) -regex="fdcache_prefetch_count ([0-9])+" -pf_count=0 -for i in $FDCACHE_FDS -do - if [[ $metrics =~ $regex ]]; then - - if [[ $(( $pf_count+$PREFETCH )) -ne ${BASH_REMATCH[1]} ]]; then - err - else - pf_count=${BASH_REMATCH[1]} - fi - - else +regex="fdcache_op_count\{op=\"prefetch_access\"\} ([0-9])+" +if [[ $metrics =~ $regex ]]; then + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then err fi -done +else + err +fi kill $PID1 wait $PID1 PID1=0 - ######### -# Test mb limit on fd and prefetch cache +# Test mb limit on fd cache ######### - rm -rf $DEBUGINFOD_CACHE_PATH rm -rf $DB env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \ - --fdcache-mbs=1 --fdcache-prefetch-mbs=1 --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \ - > vlog2$PORT1 2>&1 & + --fdcache-mbs=$FDCACHE_MBS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R > vlog2$PORT1 2>&1 & PID1=$! tempfiles vlog2$PORT1 errfiles vlog2$PORT1 @@ -139,26 +119,26 @@ wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 # All rpms need to be in the index, except the dummy permission-000 one rpms=$(find R -name \*rpm | grep -v nothing | wc -l) wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs -# Wait till both files are in the index and scan/index fully finished +kill -USR1 $PID1 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 +wait_ready $PORT1 'thread_busy{role="scan"}' 0 -archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA +# This many archives cause the fd cache to be loaded with just over 1mb of +# files. So we expect the 1mb cap off +SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1 archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA -archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA - metrics=$(curl http://127.0.0.1:$PORT1/metrics) -regex="fdcache_bytes ([0-9]+).*fdcache_prefetch_bytes ([0-9]+)" +regex="fdcache_bytes ([0-9]+)" mb=$((1024*1024)) if [[ $metrics =~ $regex ]]; then fdbytes=${BASH_REMATCH[1]} - pfbytes=${BASH_REMATCH[2]} - if [ $fdbytes -gt $mb ] || [ $pfbytes -gt $mb ]; then + if [ $fdbytes -gt $mb ] ; then err fi else -- 2.36.1