src-release.sh: Support zstd compression

  This adds -z as option to src-release.sh which adds zstd to the list
of compressors. This creates a tar.zst archive.
---
 src-release.sh | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
  

Mark Wielaard <mark@klomp.org> writes:

> This adds -z as option to src-release.sh which adds zstd to the list
> of compressors. This creates a tar.zst archive.

I don't see that much value in this - very few people use .zst for
distfiles and it's not like we'll stop producing the others for
releases.

But it's also cheap to do, so I won't bikeshed it further ;)

> ---
>  src-release.sh | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/src-release.sh b/src-release.sh
> index 0e95dde18bc3..2377c2b20ee8 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -26,6 +26,7 @@ BZIPPROG=bzip2
>  GZIPPROG=gzip
>  LZIPPROG=lzip
>  XZPROG=xz
> +ZSTDPROG=zstd
>  SHA256PROG=sha256sum
>  MAKE=make
>  CC=gcc
> @@ -241,6 +242,16 @@ do_xz()
>      $XZPROG -k -v -9 $package-$ver.tar
>  }
>  
> +# Compress the output with zstd
> +do_zstd()
> +{
> +    package=$1
> +    ver=$2
> +    echo "==> Zzipping $package-$ver.tar.zst"
> +    rm -f $package-$ver.tar.zst
> +    $ZSTDPROG -k -v -19 -T0 $package-$ver.tar
> +}
> +
>  # Compress the output with all selected compresion methods
>  do_compress()
>  {
> @@ -257,6 +268,8 @@ do_compress()
>  		do_lz $package $ver;;
>  	    xz)
>  		do_xz $package $ver;;
> +	    zstd)
> +		do_zstd $package $ver;;
>  	    *)
>  		echo "Unknown compression method: $comp" && exit 1;;
>  	esac
> @@ -352,6 +365,7 @@ usage()
>      echo "  -g: Compress with gzip"
>      echo "  -l: Compress with lzip"
>      echo "  -x: Compress with xz"
> +    echo "  -z: Compress with zstd"
>      echo "  -r <date>: Create a reproducible tarball using <date> as the mtime"
>      exit 1
>  }
> @@ -376,7 +390,7 @@ build_release()
>  
>  compressors=""
>  
> -while getopts ":bglr:x" opt; do
> +while getopts ":bglr:xz" opt; do
>      case $opt in
>  	b)
>  	    compressors="$compressors bz2";;
> @@ -388,6 +402,8 @@ while getopts ":bglr:x" opt; do
>  	    release_date=$OPTARG;;
>  	x)
>  	    compressors="$compressors xz";;
> +	z)
> +	    compressors="$compressors zstd";;
>  	\?)
>  	    echo "Invalid option: -$OPTARG" && usage;;
>    esac
  

On Mon, May 27, 2024 at 4:22 PM Sam James <sam@gentoo.org> wrote:
>
> Mark Wielaard <mark@klomp.org> writes:
>
> > This adds -z as option to src-release.sh which adds zstd to the list
> > of compressors. This creates a tar.zst archive.
>
> I don't see that much value in this - very few people use .zst for
> distfiles and it's not like we'll stop producing the others for
> releases.
>
> But it's also cheap to do, so I won't bikeshed it further ;)

Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)
Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
large and obsolete by zstd...

> > ---
> >  src-release.sh | 18 +++++++++++++++++-
> >  1 file changed, 17 insertions(+), 1 deletion(-)
> >
> > diff --git a/src-release.sh b/src-release.sh
> > index 0e95dde18bc3..2377c2b20ee8 100755
> > --- a/src-release.sh
> > +++ b/src-release.sh
> > @@ -26,6 +26,7 @@ BZIPPROG=bzip2
> >  GZIPPROG=gzip
> >  LZIPPROG=lzip
> >  XZPROG=xz
> > +ZSTDPROG=zstd
> >  SHA256PROG=sha256sum
> >  MAKE=make
> >  CC=gcc
> > @@ -241,6 +242,16 @@ do_xz()
> >      $XZPROG -k -v -9 $package-$ver.tar
> >  }
> >
> > +# Compress the output with zstd
> > +do_zstd()
> > +{
> > +    package=$1
> > +    ver=$2
> > +    echo "==> Zzipping $package-$ver.tar.zst"
> > +    rm -f $package-$ver.tar.zst
> > +    $ZSTDPROG -k -v -19 -T0 $package-$ver.tar
> > +}
> > +
> >  # Compress the output with all selected compresion methods
> >  do_compress()
> >  {
> > @@ -257,6 +268,8 @@ do_compress()
> >               do_lz $package $ver;;
> >           xz)
> >               do_xz $package $ver;;
> > +         zstd)
> > +             do_zstd $package $ver;;
> >           *)
> >               echo "Unknown compression method: $comp" && exit 1;;
> >       esac
> > @@ -352,6 +365,7 @@ usage()
> >      echo "  -g: Compress with gzip"
> >      echo "  -l: Compress with lzip"
> >      echo "  -x: Compress with xz"
> > +    echo "  -z: Compress with zstd"
> >      echo "  -r <date>: Create a reproducible tarball using <date> as the mtime"
> >      exit 1
> >  }
> > @@ -376,7 +390,7 @@ build_release()
> >
> >  compressors=""
> >
> > -while getopts ":bglr:x" opt; do
> > +while getopts ":bglr:xz" opt; do
> >      case $opt in
> >       b)
> >           compressors="$compressors bz2";;
> > @@ -388,6 +402,8 @@ while getopts ":bglr:x" opt; do
> >           release_date=$OPTARG;;
> >       x)
> >           compressors="$compressors xz";;
> > +     z)
> > +         compressors="$compressors zstd";;
> >       \?)
> >           echo "Invalid option: -$OPTARG" && usage;;
> >    esac
  

Hi,

On Mon, May 27, 2024 at 04:44:59PM -0700, Fangrui Song wrote:
> On Mon, May 27, 2024 at 4:22 PM Sam James <sam@gentoo.org> wrote:
> > Mark Wielaard <mark@klomp.org> writes:
> > > This adds -z as option to src-release.sh which adds zstd to the list
> > > of compressors. This creates a tar.zst archive.
> >
> > I don't see that much value in this - very few people use .zst for
> > distfiles and it's not like we'll stop producing the others for
> > releases.
> >
> > But it's also cheap to do, so I won't bikeshed it further ;)
> 
> Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)
> Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
> large and obsolete by zstd...

My goal was to see if .zst could/should replace the .xz archives for
the snapshots on https://snapshots.sourceware.org/ which currently
only produce .xz archives.

It could maybe also be added as alternative/extra archive for the
actual releases?

It seemed good to at least have the option to produce archives this
way.

Cheers,

Mark
  

Mark Wielaard <mark@klomp.org> writes:

> Hi,
>
> On Mon, May 27, 2024 at 04:44:59PM -0700, Fangrui Song wrote:
>> On Mon, May 27, 2024 at 4:22 PM Sam James <sam@gentoo.org> wrote:
>> > Mark Wielaard <mark@klomp.org> writes:
>> > > This adds -z as option to src-release.sh which adds zstd to the list
>> > > of compressors. This creates a tar.zst archive.
>> >
>> > I don't see that much value in this - very few people use .zst for
>> > distfiles and it's not like we'll stop producing the others for
>> > releases.
>> >
>> > But it's also cheap to do, so I won't bikeshed it further ;)
>> 
>> Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)
>> Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
>> large and obsolete by zstd...
>
> My goal was to see if .zst could/should replace the .xz archives for
> the snapshots on https://snapshots.sourceware.org/ which currently
> only produce .xz archives.
>

We don't currently support zstd everywhere on our end for distfiles so
fully replacing it would be a pain (so please keep xz as an option at
least).

Thanks!
  

Hi Mark, Hi Sam,

   I approve the patch from the binutils point of view.  So if
   the GDB folks do not object then please apply.

>>> Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)

   That was not Mark's goal. But if/when his patch goes in I will
   definitely be updating the binutils release process[1] to make
   use of it.

>>> Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
>>> large and obsolete by zstd...

   I would like to do that.  (But definitely keep .xz releases).

   Does anyone know if this will cause problems ?  I guess if people
   do complain I can always add them back to the release website.

Cheers
   Nick

[1] https://sourceware.org/git/?p=binutils-gdb.git;a=blob_plain;f=binutils/README-how-to-make-a-release;hb=HEAD
  

On 5/28/24 17:44, Nick Clifton wrote:
>   I would like to do that.  (But definitely keep .xz releases).
>

I would keep gz. It's kind of ubiquous, also on older systems.


regards,
chris
  

Nick Clifton <nickc@redhat.com> writes:

> Hi Mark, Hi Sam,
>
Hi Nick,

>   I approve the patch from the binutils point of view.  So if
>   the GDB folks do not object then please apply.
>
>>>> Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)
>
>   That was not Mark's goal. But if/when his patch goes in I will
>   definitely be updating the binutils release process[1] to make
>   use of it.
>
>>>> Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
>>>> large and obsolete by zstd...
>
>   I would like to do that.  (But definitely keep .xz releases).
>
>   Does anyone know if this will cause problems ?  I guess if people
>   do complain I can always add them back to the release website.

This plan sounds good to me - although I'd suggest we wait a bit before
dropping gzip/bzip2 in case anyone complains.

(i.e. do it in a followup)

>
> Cheers
>   Nick
>
> [1] https://sourceware.org/git/?p=binutils-gdb.git;a=blob_plain;f=binutils/README-how-to-make-a-release;hb=HEAD

thanks,
sam
  

Hi Sam,

On Tue, 2024-05-28 at 13:23 +0100, Sam James wrote:
> Mark Wielaard <mark@klomp.org> writes:
> > My goal was to see if .zst could/should replace the .xz archives for
> > the snapshots on https://snapshots.sourceware.org/ which currently
> > only produce .xz archives.
> 
> We don't currently support zstd everywhere on our end for distfiles so
> fully replacing it would be a pain (so please keep xz as an option at
> least).

OK, then we just keep xz.
Do you think full zstd support will eventually happen?

I like to at least add support for generating them because I think they
are slightly nicer than xz archives. About the same size/compression,
but using less resources, cpu at least:

$ time unxz ./binutils-2.42.50.tar.xz 

real	0m1.374s
user	0m1.289s
sys	0m0.082s

$ time unzstd ./binutils-2.42.50.tar.zst 
./binutils-2.42.50.tar.zst: 333670400 bytes                           

real	0m0.228s
user	0m0.231s
sys	0m0.082s

Admittedly it is just a second, but if you do 60 CI builds a day, then,
ehe, you saved a minute...

Cheers,

Mark
  

On Tue, May 28, 2024 at 9:13 AM Sam James <sam@gentoo.org> wrote:
>
> Nick Clifton <nickc@redhat.com> writes:
>
> > Hi Mark, Hi Sam,
> >
> Hi Nick,
>
> >   I approve the patch from the binutils point of view.  So if
> >   the GDB folks do not object then please apply.
> >
> >>>> Is the goal to add .zst to https://ftp.gnu.org/gnu/binutils/ ? :)
> >
> >   That was not Mark's goal. But if/when his patch goes in I will
> >   definitely be updating the binutils release process[1] to make
> >   use of it.
> >
> >>>> Perhaps for newer releases, .gz and .bz2 can be dropped. They are too
> >>>> large and obsolete by zstd...
> >
> >   I would like to do that.  (But definitely keep .xz releases).
> >
> >   Does anyone know if this will cause problems ?  I guess if people
> >   do complain I can always add them back to the release website.
>
> This plan sounds good to me - although I'd suggest we wait a bit before
> dropping gzip/bzip2 in case anyone complains.
>
> (i.e. do it in a followup)

Agreed.

> >
> > Cheers
> >   Nick
> >
> > [1] https://sourceware.org/git/?p=binutils-gdb.git;a=blob_plain;f=binutils/README-how-to-make-a-release;hb=HEAD
>
> thanks,
> sam

While zstd can achieve similar compression to xz, it requires a much
higher compression level, significantly slowing down the compression.
Decompression with xz is faster than bzip2, and for a 40+MiB tar.xz,
2s for decompression likely isn't a major concern for users.

.tar.gz and .tar.bz2 are not on the compression Pareto frontier. Their
presence on https://ftp.gnu.org/gnu/binutils/ is likely no longer
necessary.

The release managers can consider removing .tar.gz and .tar.bz2 for
new releases without modifying src-release.sh.
If users express a need, .tar.gz and .tar.bz2 could be temporarily
reintroduced for one release cycle. However, users should be strongly
encouraged to migrate to xz or zstd. After one additional release,
.tar.gz and .tar.bz2 can be permanently removed for new releases. Then
src-release.sh can be modified to remove support for gz and bz2.
  

Hi Mark,

> This adds -z as option to src-release.sh which adds zstd to the list
> of compressors. This creates a tar.zst archive.

I have gone ahead and applied your patch.

Nobody from the GDB community appears to have any objections, and I
cannot see any reason why anyone else might have issues with it, so
it is now in. :-)

Cheers
   Nick