From patchwork Fri Jul 19 08:32:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 94191 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 38C453860753 for ; Fri, 19 Jul 2024 08:32:56 +0000 (GMT) X-Original-To: elfutils-devel@sourceware.org Delivered-To: elfutils-devel@sourceware.org Received: from mail-oo1-xc2e.google.com (mail-oo1-xc2e.google.com [IPv6:2607:f8b0:4864:20::c2e]) by sourceware.org (Postfix) with ESMTPS id C57AC385C6CE for ; Fri, 19 Jul 2024 08:32:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C57AC385C6CE Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=osandov.com Authentication-Results: sourceware.org; spf=none smtp.mailfrom=osandov.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C57AC385C6CE Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::c2e ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1721377943; cv=none; b=Xj/4eJYlXuER2zZZvr2oG4a8lvPF0HG8I7+68R51DqrwhXib5AWeO39d9wwI0Zj30TtAQg63PAMBgMW7imkTg5LAd1ogezTIyQTfxZg4PXXi3Lsbff4/n/UkQTSyD5WHVLitlgC3RMqZ3Wk/WJ6Q67jGNzaoAmwDJG8+LyiNElo= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1721377943; c=relaxed/simple; bh=HOfAHJo///BN1FVi73kIPNB998r4isI11ggmhEvmArI=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=HHw9GjG9crs9IYzPcOO1Wy9QLBHd89I0Wib4cYg2psz0+IRGvyhPVWwq1hE111SKAShZExgU6RcRH4uSgptyJ1auvqC3gcf18HQ7tP8s525jwMJKKnRhRxN7hMz/uDk9aovR/l8tFXe/y6zROj22Garw6JaGawldjbX7jFvqegI= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-oo1-xc2e.google.com with SMTP id 006d021491bc7-5c691c6788eso831687eaf.2 for ; Fri, 19 Jul 2024 01:32:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20230601.gappssmtp.com; s=20230601; t=1721377938; x=1721982738; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u4Rj/ZnOVSL02OnzUupYCg3NbyqN5BwrWVZX7PHvJeg=; b=tTX71vV9oB31HtmbGTwC59C4K7WhGHu/8RJ8HmYJMGuhSfqhhtAXdawg+MM+hKT+48 PLRV5LzjGie6Ty6b0h1zypC0lCvqgSIn5NuyCF/JK84fDPYuFNBlAFiEzKgAZLOtuY9J OgZP6P3nyF4zyC0JnLpTlE6FGR+xUz+pbRtPjtySc0V/KseVcufmVHn+HoExGuUxO4Vs 0PZfQgqqOFpyuGm143CzsyVUMLmfSjAejxk6hlQyxPMAxBLS6K2l4eMTHxcNceZ5moef Ut+yXJ50lQLG9Yb8JULKUn6hlNHuanMCRLTce6scISvV3flkUn20KB8akcqUxI7yzSWa Hehg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721377938; x=1721982738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u4Rj/ZnOVSL02OnzUupYCg3NbyqN5BwrWVZX7PHvJeg=; b=icJSS2uX3CySDZEVBQ4RwddU11NONpdcVE4YV4y0NRTuBiSTWBCevfmYB2ITez9R+h nYzicxsBZeIN7a+kiyvp82EWTiLlA3UBsXjaLLsWz9H1FUA+v8Go2mAuAF/M//1uHvaX f8QwImeWt4iXf1G9k2DIYAsrtWZiILRTDUw5SrTLk2w+1yFBJY/7hjJa8KVJWlvO7Fxn p9+KO4CzupeDIDvHlyR35xdiyrKnQjYekcencyHB7yiG/EV16KT0ClrncVEPbg1mqeln kQTIBx/UTcMV7P6T+KATONa5LDOC0fGsdZjuMmQ8AT2hqtUh88BPcMLDPsZFbmt+IO6L Drrw== X-Gm-Message-State: AOJu0YzBI+nnGsffLfSrAi5G+ZjkcyhIOK9onLq8qBGzuXQC8+WL46zW J5u5INidUpshRb1adkYP3dBkrG84MimVEoJ2IjpAi1WnatCm+UCjhtnfS/Y6/QkeJulK7/DfF8X X X-Google-Smtp-Source: AGHT+IHkvQtCuGcznMAiz6vuPjx94FfYZCOnsO4lGhnEfZ1pHTjs/V0zDMNyDurEaOmG/LG/VjwX9Q== X-Received: by 2002:a05:6358:3423:b0:1aa:a19e:f195 with SMTP id e5c5f4694b2df-1aca9e92dc1mr481990555d.4.1721377937669; Fri, 19 Jul 2024 01:32:17 -0700 (PDT) Received: from telecaster.hsd1.wa.comcast.net ([2601:602:8980:9170::7a8e]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fd64b49467sm8832375ad.6.2024.07.19.01.32.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Jul 2024 01:32:17 -0700 (PDT) From: Omar Sandoval To: elfutils-devel@sourceware.org Cc: "Frank Ch . Eigler" , linux-debuggers@vger.kernel.org Subject: [PATCH v3 4/7] debugifod: add new table and views for seekable archives Date: Fri, 19 Jul 2024 01:32:00 -0700 Message-ID: <8354294593abfcd9cfc1f83d259921a5da0d55c3.1721377314.git.osandov@fb.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: elfutils-devel-bounces~patchwork=sourceware.org@sourceware.org From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it from the archive metadata). Add a new table, _r_seekable, keyed on the archive file id and entry file id and containing the size, offset, and mtime. It also contains the compression type just in case new seekable formats are supported in the future. In order to search this table when we get a request, we need the file ids available. Add the ids to the _query_d and _query_e views, and rename them to _query_d2 and _query_e2. This schema change is backward compatible and doesn't require reindexing. _query_d2 and _query_e2 can be renamed back the next time BUILDIDS needs to be bumped. Before this change, the database for a single kernel debuginfo RPM (kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB. This change increases that by about 70kB, only a 0.5% increase. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 24702c23..b3d80090 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -265,25 +265,39 @@ static const char DEBUGINFOD_SQLITE_DDL[] = " foreign key (content) references " BUILDIDS "_files(id) on update cascade on delete cascade,\n" " primary key (content, file, mtime)\n" " ) " WITHOUT_ROWID ";\n" + "create table if not exists " BUILDIDS "_r_seekable (\n" // seekable rpm contents + " file integer not null,\n" + " content integer not null,\n" + " type text not null,\n" + " size integer not null,\n" + " offset integer not null,\n" + " mtime integer not null,\n" + " foreign key (file) references " BUILDIDS "_files(id) on update cascade on delete cascade,\n" + " foreign key (content) references " BUILDIDS "_files(id) on update cascade on delete cascade,\n" + " primary key (file, content)\n" + " ) " WITHOUT_ROWID ";\n" // create views to glue together some of the above tables, for webapi D queries - "create view if not exists " BUILDIDS "_query_d as \n" + // NB: _query_d2 and _query_e2 were added to replace _query_d and _query_e + // without updating BUILDIDS. They can be renamed back the next time BUILDIDS + // is updated. + "create view if not exists " BUILDIDS "_query_d2 as \n" "select\n" - " b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, n.mtime as mtime, null as source1\n" + " b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as source0, n.mtime as mtime, null as id1, null as source1\n" " from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS "_f_de n\n" " where b.id = n.buildid and f0.id = n.file and n.debuginfo_p = 1\n" "union all select\n" - " b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, n.mtime as mtime, f1.name as source1\n" + " b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as source0, n.mtime as mtime, n.content as id1, f1.name as source1\n" " from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS "_files_v f1, " BUILDIDS "_r_de n\n" " where b.id = n.buildid and f0.id = n.file and f1.id = n.content and n.debuginfo_p = 1\n" ";" // ... and for E queries - "create view if not exists " BUILDIDS "_query_e as \n" + "create view if not exists " BUILDIDS "_query_e2 as \n" "select\n" - " b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, n.mtime as mtime, null as source1\n" + " b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as source0, n.mtime as mtime, null as id1, null as source1\n" " from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS "_f_de n\n" " where b.id = n.buildid and f0.id = n.file and n.executable_p = 1\n" "union all select\n" - " b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, n.mtime as mtime, f1.name as source1\n" + " b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as source0, n.mtime as mtime, n.content as id1, f1.name as source1\n" " from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS "_files_v f1, " BUILDIDS "_r_de n\n" " where b.id = n.buildid and f0.id = n.file and f1.id = n.content and n.executable_p = 1\n" ";" @@ -2557,7 +2571,7 @@ handle_buildid (MHD_Connection* conn, if (atype_code == "D") { pp = new sqlite_ps (thisdb, "mhd-query-d", - "select mtime, sourcetype, source0, source1 from " BUILDIDS "_query_d where buildid = ? " + "select mtime, sourcetype, source0, source1 from " BUILDIDS "_query_d2 where buildid = ? " "order by mtime desc"); pp->reset(); pp->bind(1, buildid); @@ -2565,7 +2579,7 @@ handle_buildid (MHD_Connection* conn, else if (atype_code == "E") { pp = new sqlite_ps (thisdb, "mhd-query-e", - "select mtime, sourcetype, source0, source1 from " BUILDIDS "_query_e where buildid = ? " + "select mtime, sourcetype, source0, source1 from " BUILDIDS "_query_e2 where buildid = ? " "order by mtime desc"); pp->reset(); pp->bind(1, buildid); @@ -2589,9 +2603,9 @@ handle_buildid (MHD_Connection* conn, else if (atype_code == "I") { pp = new sqlite_ps (thisdb, "mhd-query-i", - "select mtime, sourcetype, source0, source1, 1 as debug_p from " BUILDIDS "_query_d where buildid = ? " + "select mtime, sourcetype, source0, source1, 1 as debug_p from " BUILDIDS "_query_d2 where buildid = ? " "union all " - "select mtime, sourcetype, source0, source1, 0 as debug_p from " BUILDIDS "_query_e where buildid = ? " + "select mtime, sourcetype, source0, source1, 0 as debug_p from " BUILDIDS "_query_e2 where buildid = ? " "order by debug_p desc, mtime desc"); pp->reset(); pp->bind(1, buildid);