libsanitizer: merge from upstream

  Hello.

I'm going to do merge from upstream.

Patch can bootstrap on x86_64-linux-gnu and survives regression tests. I've also tested
on ppc64le-linux-gnu and verified the ABI.

The only real change is a small change in
gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c
where we need --param=asan-use-after-return=0.

I'm going to push the patches.

Thanks,
Martin
  

On Wed, May 4, 2022 at 1:59 AM Martin Liška <mliska@suse.cz> wrote:
>
> Hello.
>
> I'm going to do merge from upstream.
>
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. I've also tested
> on ppc64le-linux-gnu and verified the ABI.
>
> The only real change is a small change in
> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c
> where we need --param=asan-use-after-return=0.
>
> I'm going to push the patches.

Hi,

I am checking in this patch to cherry-pick

f52e365092aa [sanitizer] Use newfstatat for x32

to restore x32 build.
  

On 5/5/22 01:07, H.J. Lu wrote:
> On Wed, May 4, 2022 at 1:59 AM Martin Liška <mliska@suse.cz> wrote:
>> 
>> Hello.
>> 
>> I'm going to do merge from upstream.
>> 
>> Patch can bootstrap on x86_64-linux-gnu and survives regression
>> tests. I've also tested on ppc64le-linux-gnu and verified the ABI.
>> 
>> The only real change is a small change in 
>> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c where we
>> need --param=asan-use-after-return=0.
>> 
>> I'm going to push the patches.
> 
> Hi,
> 
> I am checking in this patch to cherry-pick
> 
> f52e365092aa [sanitizer] Use newfstatat for x32
> 
> to restore x32 build.
> 

I'm going to do one more merge from upstream
(75f9e83ace52773af65dcebca543005ec8a2705d) as we want to include Tobias's
revision 6f095babc2b7d564168c7afc5bf6afb2188fd6b4 and my
revision f1b9245199f3457a4d06d32d1bc6e44573c166e3.

Martin
  

On Thu, May 5, 2022 at 4:24 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/5/22 01:07, H.J. Lu wrote:
> > On Wed, May 4, 2022 at 1:59 AM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> Hello.
> >>
> >> I'm going to do merge from upstream.
> >>
> >> Patch can bootstrap on x86_64-linux-gnu and survives regression
> >> tests. I've also tested on ppc64le-linux-gnu and verified the ABI.
> >>
> >> The only real change is a small change in
> >> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c where we
> >> need --param=asan-use-after-return=0.
> >>
> >> I'm going to push the patches.
> >
> > Hi,
> >
> > I am checking in this patch to cherry-pick
> >
> > f52e365092aa [sanitizer] Use newfstatat for x32
> >
> > to restore x32 build.
> >
>
> I'm going to do one more merge from upstream
> (75f9e83ace52773af65dcebca543005ec8a2705d) as we want to include Tobias's
> revision 6f095babc2b7d564168c7afc5bf6afb2188fd6b4 and my
> revision f1b9245199f3457a4d06d32d1bc6e44573c166e3.

I am testing a patch for

https://github.com/llvm/llvm-project/issues/55288

to fix:

https://gcc.gnu.org/pipermail/gcc-regression/2022-May/076571.html

The same bug is also in GCC 12.  But somehow, it doesn't show up in
GCC tests.
  

On 5/5/22 18:21, H.J. Lu wrote:
> On Thu, May 5, 2022 at 4:24 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 5/5/22 01:07, H.J. Lu wrote:
>>> On Wed, May 4, 2022 at 1:59 AM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> Hello.
>>>>
>>>> I'm going to do merge from upstream.
>>>>
>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression
>>>> tests. I've also tested on ppc64le-linux-gnu and verified the ABI.
>>>>
>>>> The only real change is a small change in
>>>> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c where we
>>>> need --param=asan-use-after-return=0.
>>>>
>>>> I'm going to push the patches.
>>>
>>> Hi,
>>>
>>> I am checking in this patch to cherry-pick
>>>
>>> f52e365092aa [sanitizer] Use newfstatat for x32
>>>
>>> to restore x32 build.
>>>
>>
>> I'm going to do one more merge from upstream
>> (75f9e83ace52773af65dcebca543005ec8a2705d) as we want to include Tobias's
>> revision 6f095babc2b7d564168c7afc5bf6afb2188fd6b4 and my
>> revision f1b9245199f3457a4d06d32d1bc6e44573c166e3.
> 
> I am testing a patch for
> 
> https://github.com/llvm/llvm-project/issues/55288
> 
> to fix:
> 
> https://gcc.gnu.org/pipermail/gcc-regression/2022-May/076571.html

Interesting. How did you run these tests that the error shows up?

> 
> The same bug is also in GCC 12.  But somehow, it doesn't show up in
> GCC tests.

So please backport it once it's merged.

Martin
  

On Thu, May 5, 2022 at 11:28 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/5/22 18:21, H.J. Lu wrote:
> > On Thu, May 5, 2022 at 4:24 AM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 5/5/22 01:07, H.J. Lu wrote:
> >>> On Wed, May 4, 2022 at 1:59 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>
> >>>> Hello.
> >>>>
> >>>> I'm going to do merge from upstream.
> >>>>
> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression
> >>>> tests. I've also tested on ppc64le-linux-gnu and verified the ABI.
> >>>>
> >>>> The only real change is a small change in
> >>>> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c where we
> >>>> need --param=asan-use-after-return=0.
> >>>>
> >>>> I'm going to push the patches.
> >>>
> >>> Hi,
> >>>
> >>> I am checking in this patch to cherry-pick
> >>>
> >>> f52e365092aa [sanitizer] Use newfstatat for x32
> >>>
> >>> to restore x32 build.
> >>>
> >>
> >> I'm going to do one more merge from upstream
> >> (75f9e83ace52773af65dcebca543005ec8a2705d) as we want to include Tobias's
> >> revision 6f095babc2b7d564168c7afc5bf6afb2188fd6b4 and my
> >> revision f1b9245199f3457a4d06d32d1bc6e44573c166e3.
> >
> > I am testing a patch for
> >
> > https://github.com/llvm/llvm-project/issues/55288

I submitted:

https://reviews.llvm.org/D125025

> > to fix:
> >
> > https://gcc.gnu.org/pipermail/gcc-regression/2022-May/076571.html
>
> Interesting. How did you run these tests that the error shows up?

Just normal GCC bootstrap and check with x32 enabled.

> >
> > The same bug is also in GCC 12.  But somehow, it doesn't show up in
> > GCC tests.
>
> So please backport it once it's merged.
>

Will do after GCC 12 is released.

Thanks.
  

On 5/4/22 10:59, Martin Liška wrote:
> Hello.
> 
> I'm going to do merge from upstream.
> 
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. I've also tested
> on ppc64le-linux-gnu and verified the ABI.
> 
> The only real change is a small change in
> gcc/testsuite/c-c++-common/asan/alloca_loop_unpoisoning.c
> where we need --param=asan-use-after-return=0.
> 
> I'm going to push the patches.
> 
> Thanks,
> Martin

Hi.

I've just done one more merge from upstream.
Upstream revision: 84a71d5259c2682403cdbd8710592410a2f128ab.

Cheers,
Martin
  

Hi Martin,

> On 30 Aug 2022, at 11:53, Martin Liška <mliska@suse.cz> wrote:
> 
> On 5/4/22 10:59, Martin Liška wrote:
>> Hello.
> I've just done one more merge from upstream.
> Upstream revision: 84a71d5259c2682403cdbd8710592410a2f128ab.

Which (again) breaks bootstrap on Darwin (since upstream uses features
and APIs not present in GCC and/or supported by the systems we support).

Worked around as below; we will need to find a way to handle the
macOS 13+ changes.

Iain


[pushed] [libsanitizer, Darwin] Fix bootstrap after recent merge.

The latest merge to libsanitizer includes changes to handle macOS 13+.
However, these changes are incompatible with GCC and so we need to find
an alternate solution.  To restore bootstrap back this change out until
the alternate can be found.
---
 .../sanitizer_procmaps_mac.cpp                | 62 +++----------------
 1 file changed, 10 insertions(+), 52 deletions(-)

diff --git a/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp b/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
index 4b0e6781976..ba4259acd46 100644
--- a/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
+++ b/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
@@ -146,8 +146,13 @@ static bool IsDyldHdr(const mach_header *hdr) {
 // until we hit a Mach header matching dyld instead. These recurse
 // calls are expensive, but the first memory map generation occurs
 // early in the process, when dyld is one of the only images loaded,
-// so it will be hit after only a few iterations.  These assumptions don't hold
-// on macOS 13+ anymore (dyld itself has moved into the shared cache).
+// so it will be hit after only a few iterations.  These assumptions don't
+// hold on macOS 13+ anymore (dyld itself has moved into the shared cache).
+
+// FIXME: Unfortunately, the upstream revised version to deal with macOS 13+
+// is incompatible with GCC and also uses APIs not available on earlier
+// systems which we support; backed out for now.
+
 static mach_header *GetDyldImageHeaderViaVMRegion() {
   vm_address_t address = 0;
 
@@ -171,64 +176,17 @@ static mach_header *GetDyldImageHeaderViaVMRegion() {
   }
 }
 
-extern "C" {
-struct dyld_shared_cache_dylib_text_info {
-  uint64_t version;  // current version 2
-  // following fields all exist in version 1
-  uint64_t loadAddressUnslid;
-  uint64_t textSegmentSize;
-  uuid_t dylibUuid;
-  const char *path;  // pointer invalid at end of iterations
-  // following fields all exist in version 2
-  uint64_t textSegmentOffset;  // offset from start of cache
-};
-typedef struct dyld_shared_cache_dylib_text_info
-    dyld_shared_cache_dylib_text_info;
-
-extern bool _dyld_get_shared_cache_uuid(uuid_t uuid);
-extern const void *_dyld_get_shared_cache_range(size_t *length);
-extern int dyld_shared_cache_iterate_text(
-    const uuid_t cacheUuid,
-    void (^callback)(const dyld_shared_cache_dylib_text_info *info));
-}  // extern "C"
-
-static mach_header *GetDyldImageHeaderViaSharedCache() {
-  uuid_t uuid;
-  bool hasCache = _dyld_get_shared_cache_uuid(uuid);
-  if (!hasCache)
-    return nullptr;
-
-  size_t cacheLength;
-  __block uptr cacheStart = (uptr)_dyld_get_shared_cache_range(&cacheLength);
-  CHECK(cacheStart && cacheLength);
-
-  __block mach_header *dyldHdr = nullptr;
-  int res = dyld_shared_cache_iterate_text(
-      uuid, ^(const dyld_shared_cache_dylib_text_info *info) {
-        CHECK_GE(info->version, 2);
-        mach_header *hdr =
-            (mach_header *)(cacheStart + info->textSegmentOffset);
-        if (IsDyldHdr(hdr))
-          dyldHdr = hdr;
-      });
-  CHECK_EQ(res, 0);
-
-  return dyldHdr;
-}
-
 const mach_header *get_dyld_hdr() {
   if (!dyld_hdr) {
     // On macOS 13+, dyld itself has moved into the shared cache.  Looking it up
     // via vm_region_recurse_64() causes spins/hangs/crashes.
+    // FIXME: find a way to do this compatible with GCC.
     if (GetMacosAlignedVersion() >= MacosVersion(13, 0)) {
-      dyld_hdr = GetDyldImageHeaderViaSharedCache();
-      if (!dyld_hdr) {
         VReport(1,
-                "Failed to lookup the dyld image header in the shared cache on "
-                "macOS 13+ (or no shared cache in use).  Falling back to "
+                "looking up the dyld image header in the shared cache on "
+                "macOS 13+ is not yet supported.  Falling back to "
                 "lookup via vm_region_recurse_64().\n");
         dyld_hdr = GetDyldImageHeaderViaVMRegion();
-      }
     } else {
       dyld_hdr = GetDyldImageHeaderViaVMRegion();
     }
  

On 9/4/22 21:50, Iain Sandoe wrote:
> Hi Martin,
> 
>> On 30 Aug 2022, at 11:53, Martin Liška <mliska@suse.cz> wrote:
>>
>> On 5/4/22 10:59, Martin Liška wrote:
>>> Hello.
>> I've just done one more merge from upstream.
>> Upstream revision: 84a71d5259c2682403cdbd8710592410a2f128ab.
> 
> Which (again) breaks bootstrap on Darwin (since upstream uses features
> and APIs not present in GCC and/or supported by the systems we support).

Hi.

Can you please report that to upstream and create a patch that would
be accepted by upstream?

Thanks,
Martin

> 
> Worked around as below; we will need to find a way to handle the
> macOS 13+ changes.
> 
> Iain
> 
> 
> [pushed] [libsanitizer, Darwin] Fix bootstrap after recent merge.
> 
> The latest merge to libsanitizer includes changes to handle macOS 13+.
> However, these changes are incompatible with GCC and so we need to find
> an alternate solution.  To restore bootstrap back this change out until
> the alternate can be found.
> ---
>  .../sanitizer_procmaps_mac.cpp                | 62 +++----------------
>  1 file changed, 10 insertions(+), 52 deletions(-)
> 
> diff --git a/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp b/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
> index 4b0e6781976..ba4259acd46 100644
> --- a/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
> +++ b/libsanitizer/sanitizer_common/sanitizer_procmaps_mac.cpp
> @@ -146,8 +146,13 @@ static bool IsDyldHdr(const mach_header *hdr) {
>  // until we hit a Mach header matching dyld instead. These recurse
>  // calls are expensive, but the first memory map generation occurs
>  // early in the process, when dyld is one of the only images loaded,
> -// so it will be hit after only a few iterations.  These assumptions don't hold
> -// on macOS 13+ anymore (dyld itself has moved into the shared cache).
> +// so it will be hit after only a few iterations.  These assumptions don't
> +// hold on macOS 13+ anymore (dyld itself has moved into the shared cache).
> +
> +// FIXME: Unfortunately, the upstream revised version to deal with macOS 13+
> +// is incompatible with GCC and also uses APIs not available on earlier
> +// systems which we support; backed out for now.
> +
>  static mach_header *GetDyldImageHeaderViaVMRegion() {
>    vm_address_t address = 0;
>  
> @@ -171,64 +176,17 @@ static mach_header *GetDyldImageHeaderViaVMRegion() {
>    }
>  }
>  
> -extern "C" {
> -struct dyld_shared_cache_dylib_text_info {
> -  uint64_t version;  // current version 2
> -  // following fields all exist in version 1
> -  uint64_t loadAddressUnslid;
> -  uint64_t textSegmentSize;
> -  uuid_t dylibUuid;
> -  const char *path;  // pointer invalid at end of iterations
> -  // following fields all exist in version 2
> -  uint64_t textSegmentOffset;  // offset from start of cache
> -};
> -typedef struct dyld_shared_cache_dylib_text_info
> -    dyld_shared_cache_dylib_text_info;
> -
> -extern bool _dyld_get_shared_cache_uuid(uuid_t uuid);
> -extern const void *_dyld_get_shared_cache_range(size_t *length);
> -extern int dyld_shared_cache_iterate_text(
> -    const uuid_t cacheUuid,
> -    void (^callback)(const dyld_shared_cache_dylib_text_info *info));
> -}  // extern "C"
> -
> -static mach_header *GetDyldImageHeaderViaSharedCache() {
> -  uuid_t uuid;
> -  bool hasCache = _dyld_get_shared_cache_uuid(uuid);
> -  if (!hasCache)
> -    return nullptr;
> -
> -  size_t cacheLength;
> -  __block uptr cacheStart = (uptr)_dyld_get_shared_cache_range(&cacheLength);
> -  CHECK(cacheStart && cacheLength);
> -
> -  __block mach_header *dyldHdr = nullptr;
> -  int res = dyld_shared_cache_iterate_text(
> -      uuid, ^(const dyld_shared_cache_dylib_text_info *info) {
> -        CHECK_GE(info->version, 2);
> -        mach_header *hdr =
> -            (mach_header *)(cacheStart + info->textSegmentOffset);
> -        if (IsDyldHdr(hdr))
> -          dyldHdr = hdr;
> -      });
> -  CHECK_EQ(res, 0);
> -
> -  return dyldHdr;
> -}
> -
>  const mach_header *get_dyld_hdr() {
>    if (!dyld_hdr) {
>      // On macOS 13+, dyld itself has moved into the shared cache.  Looking it up
>      // via vm_region_recurse_64() causes spins/hangs/crashes.
> +    // FIXME: find a way to do this compatible with GCC.
>      if (GetMacosAlignedVersion() >= MacosVersion(13, 0)) {
> -      dyld_hdr = GetDyldImageHeaderViaSharedCache();
> -      if (!dyld_hdr) {
>          VReport(1,
> -                "Failed to lookup the dyld image header in the shared cache on "
> -                "macOS 13+ (or no shared cache in use).  Falling back to "
> +                "looking up the dyld image header in the shared cache on "
> +                "macOS 13+ is not yet supported.  Falling back to "
>                  "lookup via vm_region_recurse_64().\n");
>          dyld_hdr = GetDyldImageHeaderViaVMRegion();
> -      }
>      } else {
>        dyld_hdr = GetDyldImageHeaderViaVMRegion();
>      }
  

Hi Martin,

> On 9/4/22 21:50, Iain Sandoe wrote:
>> Hi Martin,
>> 
>>> On 30 Aug 2022, at 11:53, Martin Liška <mliska@suse.cz> wrote:
>>>
>>> On 5/4/22 10:59, Martin Liška wrote:
>>>> Hello.
>>> I've just done one more merge from upstream.
>>> Upstream revision: 84a71d5259c2682403cdbd8710592410a2f128ab.
>> 
>> Which (again) breaks bootstrap on Darwin (since upstream uses features
>> and APIs not present in GCC and/or supported by the systems we support).
>
> Hi.
>
> Can you please report that to upstream and create a patch that would
> be accepted by upstream?

this may be difficult, unfortunately.  Some time ago, I tried to get an
answer to the question which macOS versions are supposed to be supported
by upstream LLVM and got nothing, despite the massive involvement of
Apple in the project.  Still, it's certainly worth a try.

As a datapoint, both recent libsanitizer imports broke Solaris 11.3
bootstrap, which is still supported by GCC, but never was by LLVM.  In
the first case, I managed to get a patch in to fix that (and a
non-trivial one at that), in the current case I even created the
breakage myself by (again) forgetting that upstream eventually gets
imported into GCC with its wider range of supported Solaris versions.

	Rainer