Replace usage of ar with stored library manifest files

Message ID 20250113235601.6829-1-mcpratt@pm.me
State Committed
Headers
Series Replace usage of ar with stored library manifest files |

Commit Message

Michael Pratt Jan. 13, 2025, 11:56 p.m. UTC
  The ar program is called to assemble a list of objects
within each archive to assist in building combined libraries,
however make already has this information
when processing the subdirectory for that respective library.
The list can be saved in a "manifest" file
instead of being generated whenever it is needed
for use with other subdirectories.

Even though the difference in time is insignificant,
a simple "echo" and "cat" is as much as 10 times faster
than a call to "ar t" for printing the archive members.

Since elfutils builds ar,
this also removes the awkward circular dependency
where an installation of ar is required
to build the libraries for ar.

Additionally, not every version of ar is equally portable,
as native versions of ar on macOS and other BSD-like distributions
may print out a special archive member like "__.SYMDEF"
which is not a compiled object but rather just metadata
from ranlib, leading to a build failure.

Avoid these limitations by removing usage of ar
and adding build and clean rules
for the usage of archive manifest files.

    * .gitignore: exclude ".manifest" file extension.
    * backends/Makefile.am: add manifest file build and clean rules.
    * debuginfod/Makefile.am: Likewise.
    * lib/Makefile.am: Likewise.
    * libasm/Makefile.am: Likewise.
    * libcpu/Makefile.am: Likewise.
    * libdw/Makefile.am: Likewise,
      and set object lists to manifest contents.
    * libdwelf/Makefile.am: Likewise.
    * libdwfl/Makefile.am: Likewise.
    * libebl/Makefile.am: Likewise.
    * libelf/Makefile.am: Likewise,
      and set object lists to manifest contents.
    * src/Makefile.am: Likewise.

Signed-off-by: Michael Pratt <mcpratt@pm.me>
---
 .gitignore             |  1 +
 backends/Makefile.am   |  6 ++++++
 debuginfod/Makefile.am |  7 ++++++-
 lib/Makefile.am        |  7 +++++++
 libasm/Makefile.am     |  7 ++++++-
 libcpu/Makefile.am     |  7 ++++++-
 libdw/Makefile.am      | 17 +++++++++++------
 libdwelf/Makefile.am   |  6 ++++++
 libdwfl/Makefile.am    |  6 ++++++
 libebl/Makefile.am     |  6 ++++++
 libelf/Makefile.am     |  9 +++++++--
 src/Makefile.am        |  7 ++++++-
 12 files changed, 74 insertions(+), 12 deletions(-)
  

Comments

Mark Wielaard Jan. 21, 2025, 6:25 p.m. UTC | #1
Hi Michael,

On Mon, 2025-01-13 at 23:56 +0000, Michael Pratt wrote:
> The ar program is called to assemble a list of objects
> within each archive to assist in building combined libraries,
> however make already has this information
> when processing the subdirectory for that respective library.
> The list can be saved in a "manifest" file
> instead of being generated whenever it is needed
> for use with other subdirectories.
> 
> Even though the difference in time is insignificant,
> a simple "echo" and "cat" is as much as 10 times faster
> than a call to "ar t" for printing the archive members.
> 
> Since elfutils builds ar,
> this also removes the awkward circular dependency
> where an installation of ar is required
> to build the libraries for ar.
> 
> Additionally, not every version of ar is equally portable,
> as native versions of ar on macOS and other BSD-like distributions
> may print out a special archive member like "__.SYMDEF"
> which is not a compiled object but rather just metadata
> from ranlib, leading to a build failure.
> 
> Avoid these limitations by removing usage of ar
> and adding build and clean rules
> for the usage of archive manifest files.

Even though I wasn't even aware of these limitations (I don't think
anybody ever reported running into them), I really like this change.
And the implementation is also very clean.

>     * .gitignore: exclude ".manifest" file extension.
>     * backends/Makefile.am: add manifest file build and clean rules.
>     * debuginfod/Makefile.am: Likewise.
>     * lib/Makefile.am: Likewise.
>     * libasm/Makefile.am: Likewise.
>     * libcpu/Makefile.am: Likewise.
>     * libdw/Makefile.am: Likewise,
>       and set object lists to manifest contents.
>     * libdwelf/Makefile.am: Likewise.
>     * libdwfl/Makefile.am: Likewise.
>     * libebl/Makefile.am: Likewise.
>     * libelf/Makefile.am: Likewise,
>       and set object lists to manifest contents.
>     * src/Makefile.am: Likewise.

Pushed as is.

Thanks,

Mark
  

Patch

diff --git a/.gitignore b/.gitignore
index 8bcd88d7..b21d7490 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,7 @@ 
 *.a
 *.gcda
 *.gcno
+*.manifest
 *.o
 *.orig
 *.os
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 347dad52..540d0c6c 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -123,4 +123,10 @@  noinst_HEADERS = libebl_CPU.h common-reloc.c linux-core-note.c x86_corenote.c
 
 EXTRA_DIST = $(modules:=_reloc.def)
 
+EXTRA_libebl_backends_a_DEPENDENCIES = libebl_backends.manifest
+
+libebl_backends.manifest: $(libebl_backends_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libebl_backends_a_OBJECTS) $(am_libebl_backends_pic_a_OBJECTS)
+CLEANFILES = $(EXTRA_libebl_backends_a_DEPENDENCIES)
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index 45b5339f..4727e5fa 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -126,8 +126,13 @@  endif
 
 EXTRA_DIST = libdebuginfod.map
 
+EXTRA_libdebuginfod_a_DEPENDENCIES = libdebuginfod.manifest
+
+libdebuginfod.manifest: $(libdebuginfod_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libdebuginfod_a_OBJECTS) $(am_libdebuginfod_pic_a_OBJECTS) $(LIBDEBUGINFOD_SONAME)
-CLEANFILES = libdebuginfod.so
+CLEANFILES = libdebuginfod.so $(EXTRA_libdebuginfod_a_DEPENDENCIES)
 
 # automake std-options override: arrange to pass LD_LIBRARY_PATH
 installcheck-binPROGRAMS: $(bin_PROGRAMS)
diff --git a/lib/Makefile.am b/lib/Makefile.am
index 45b351c1..bf5f9ee2 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -43,3 +43,10 @@  noinst_HEADERS = fixedsizehash.h libeu.h system.h dynamicsizehash.h list.h \
 		 eu-search.h locks.h
 
 EXTRA_DIST = dynamicsizehash.c dynamicsizehash_concurrent.c
+
+EXTRA_libeu_a_DEPENDENCIES = libeu.manifest
+
+libeu.manifest: $(libeu_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
+CLEANFILES = $(EXTRA_libeu_a_DEPENDENCIES)
diff --git a/libasm/Makefile.am b/libasm/Makefile.am
index 324fd095..969db935 100644
--- a/libasm/Makefile.am
+++ b/libasm/Makefile.am
@@ -87,5 +87,10 @@  noinst_HEADERS = libasmP.h symbolhash.h
 
 EXTRA_DIST = libasm.map
 
+EXTRA_libasm_a_DEPENDENCIES = libasm.manifest
+
+libasm.manifest: $(libasm_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libasm_a_OBJECTS) $(am_libasm_pic_a_OBJECTS) libasm.so.$(VERSION)
-CLEANFILES = libasm.so
+CLEANFILES = libasm.so $(EXTRA_libasm_a_DEPENDENCIES)
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index 2e87021b..df16439c 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -103,6 +103,11 @@  bpf_disasm_CFLAGS = -Wno-format-nonliteral
 
 EXTRA_DIST = defs/i386
 
+EXTRA_libcpu_a_DEPENDENCIES = libcpu.manifest
+
+libcpu.manifest: $(libcpu_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libcpu_a_OBJECTS) $(am_libcpu_pic_a_OBJECTS) $(i386_gendis_OBJECTS)
-CLEANFILES = $(foreach P,i386 x86_64,$P_defs $P.mnemonics)
+CLEANFILES = $(foreach P,i386 x86_64,$P_defs $P.mnemonics) $(EXTRA_libcpu_a_DEPENDENCIES)
 MAINTAINERCLEANFILES = $(foreach P,i386 x86_64, $P_defs $P_dis.h $P_parse.h)
diff --git a/libdw/Makefile.am b/libdw/Makefile.am
index 62f4359e..8cbe0f32 100644
--- a/libdw/Makefile.am
+++ b/libdw/Makefile.am
@@ -132,19 +132,19 @@  uninstall: uninstall-am
 	rm -f $(DESTDIR)$(libdir)/libdw.so
 	rmdir --ignore-fail-on-non-empty $(DESTDIR)$(includedir)/elfutils
 
-libdwfl_objects = $(shell $(AR) t ../libdwfl/libdwfl.a)
+libdwfl_objects = $(shell cat ../libdwfl/libdwfl.manifest)
 libdw_a_LIBADD = $(addprefix ../libdwfl/,$(libdwfl_objects))
 
-libdwelf_objects = $(shell $(AR) t ../libdwelf/libdwelf.a)
+libdwelf_objects = $(shell cat ../libdwelf/libdwelf.manifest)
 libdw_a_LIBADD += $(addprefix ../libdwelf/,$(libdwelf_objects))
 
-libebl_objects = $(shell $(AR) t ../libebl/libebl.a)
+libebl_objects = $(shell cat ../libebl/libebl.manifest)
 libdw_a_LIBADD += $(addprefix ../libebl/,$(libebl_objects))
 
-backends_objects = $(shell $(AR) t ../backends/libebl_backends.a)
+backends_objects = $(shell cat ../backends/libebl_backends.manifest)
 libdw_a_LIBADD += $(addprefix ../backends/,$(backends_objects))
 
-libcpu_objects = $(shell $(AR) t ../libcpu/libcpu.a)
+libcpu_objects = $(shell cat ../libcpu/libcpu.manifest)
 libdw_a_LIBADD += $(addprefix ../libcpu/,$(libcpu_objects))
 
 noinst_HEADERS = libdwP.h memory-access.h dwarf_abbrev_hash.h \
@@ -152,6 +152,11 @@  noinst_HEADERS = libdwP.h memory-access.h dwarf_abbrev_hash.h \
 
 EXTRA_DIST = libdw.map
 
+EXTRA_libdw_a_DEPENDENCIES = libdw.manifest
+
+libdw.manifest: $(libdw_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libdw_a_OBJECTS) $(am_libdw_pic_a_OBJECTS) libdw.so.$(VERSION)
-CLEANFILES = libdw.so
+CLEANFILES = libdw.so $(EXTRA_libdw_a_DEPENDENCIES)
 MAINTAINERCLEANFILES = $(srcdir)/known-dwarf.h
diff --git a/libdwelf/Makefile.am b/libdwelf/Makefile.am
index 30abdf81..5fb57379 100644
--- a/libdwelf/Makefile.am
+++ b/libdwelf/Makefile.am
@@ -54,4 +54,10 @@  libeu = ../lib/libeu.a
 libdwelf_pic_a_SOURCES =
 am_libdwelf_pic_a_OBJECTS = $(libdwelf_a_SOURCES:.c=.os)
 
+EXTRA_libdwelf_a_DEPENDENCIES = libdwelf.manifest
+
+libdwelf.manifest: $(libdwelf_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libdwelf_a_OBJECTS) $(am_libdwelf_pic_a_OBJECTS)
+CLEANFILES = $(EXTRA_libdwelf_a_DEPENDENCIES)
diff --git a/libdwfl/Makefile.am b/libdwfl/Makefile.am
index 171bb368..b30b86f0 100644
--- a/libdwfl/Makefile.am
+++ b/libdwfl/Makefile.am
@@ -94,4 +94,10 @@  am_libdwfl_pic_a_OBJECTS = $(libdwfl_a_SOURCES:.c=.os)
 
 noinst_HEADERS = libdwflP.h
 
+EXTRA_libdwfl_a_DEPENDENCIES = libdwfl.manifest
+
+libdwfl.manifest: $(libdwfl_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libdwfl_a_OBJECTS) $(am_libdwfl_pic_a_OBJECTS)
+CLEANFILES = $(EXTRA_libdwfl_a_DEPENDENCIES)
diff --git a/libebl/Makefile.am b/libebl/Makefile.am
index e94eaa7f..ea092b5a 100644
--- a/libebl/Makefile.am
+++ b/libebl/Makefile.am
@@ -61,4 +61,10 @@  am_libebl_pic_a_OBJECTS = $(libebl_a_SOURCES:.c=.os)
 
 noinst_HEADERS = libebl.h libeblP.h ebl-hooks.h
 
+EXTRA_libebl_a_DEPENDENCIES = libebl.manifest
+
+libebl.manifest: $(libebl_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libebl_a_OBJECTS) $(am_libebl_pic_a_OBJECTS)
+CLEANFILES = $(EXTRA_libebl_a_DEPENDENCIES)
diff --git a/libelf/Makefile.am b/libelf/Makefile.am
index e91bcd6e..05484c12 100644
--- a/libelf/Makefile.am
+++ b/libelf/Makefile.am
@@ -122,7 +122,7 @@  libelf.so: $(srcdir)/libelf.map $(libelf_so_LIBS) $(libelf_so_DEPS)
 	@$(textrel_check)
 	$(AM_V_at)ln -fs $@ $@.$(VERSION)
 
-libeu_objects = $(shell $(AR) t ../lib/libeu.a)
+libeu_objects = $(shell cat ../lib/libeu.manifest)
 libelf_a_LIBADD = $(addprefix ../lib/,$(libeu_objects))
 
 install: install-am libelf.so
@@ -138,5 +138,10 @@  uninstall: uninstall-am
 
 EXTRA_DIST = libelf.map
 
+EXTRA_libelf_a_DEPENDENCIES = libelf.manifest
+
+libelf.manifest: $(libelf_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = $(am_libelf_a_OBJECTS) $(am_libelf_pic_a_OBJECTS) libelf.so.$(VERSION)
-CLEANFILES = libelf.so
+CLEANFILES = libelf.so $(EXTRA_libelf_a_DEPENDENCIES)
diff --git a/src/Makefile.am b/src/Makefile.am
index 97a0c61a..ed245fc1 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -43,8 +43,13 @@  bin_SCRIPTS = make-debug-archive
 
 EXTRA_DIST = arlib.h debugpred.h make-debug-archive.in
 
+EXTRA_libar_a_DEPENDENCIES = libar.manifest
+
+libar.manifest: $(libar_a_OBJECTS)
+	$(AM_V_GEN)echo $^ > $@
+
 MOSTLYCLEANFILES = *.gconv
-CLEANFILES = $(bin_SCRIPTS)
+CLEANFILES = $(bin_SCRIPTS) $(EXTRA_libar_a_DEPENDENCIES)
 
 if BUILD_STATIC
 libasm = ../libasm/libasm.a