building GLIBC with -fsanitize=address

  Hello,

I want to ask some assistance with building the GLIBC with AddressSanitizer
(aka asan, GCC's -fsanitize=address flag).
The end goal is to have libc-asan.so, a shared libc library with code
instrumented by asan.

Simply doing CFLAGS='-g -O1 -fsanitize=address' ../glibc/configure ..
does not work -- it fails at configure time with
configure: error: Need linker with .init_array/.fini_array support.
Even if we bypass this error there will be others and, finally, we
don't want to instrument
every file we build (e.g. there are object files that go into the
ld.so which we don't want to instrument).

One of the things we discussed at the Cauldron was to add a configure
option (e.g. --enable-asan)
similar to the existing --enable-profile which adds the -pg build flag.
This is indeed similar with one major difference: --enable-profile
builds a static libc and
for asan we need a DSO. I've managed to make a patch that builds
libc-asan.a (attached)
but I don't see how to get libc-asan.so from it.

Thoughts?

Thanks,

--kcc
  

The machinery based on object-suffixes takes care of having separate object
file builds and archive libraries.  But if you want to make a separate
flavor of shared objects from those, you'll need to duplicate (refactor)
a bunch more rules for doing the actual ld -shared step.

I'm inclined to think that this is not really the right model for an ASan
build.  But to be sure, we need to dig deeper into exactly what you/we
envision an ASan build containing and how it would be used.

The tack you've attempted leads to the single canonical glibc build
(optionally) producing another library variant.  The alternate approach is
a configure switch that changes what the "primary" build does rather than
adding a variant.  That is, you would configure with --enable-asan (or
perhaps this should be --with-asan for this approach) as an orthogonal
switch to --{en,dis}able-{static,shared,profile}.  This build would produce
a libc.so (and libc.a too unless --disable-static) and ld.so built with
-fsanitize=address.  You'd configure this for a different --prefix or
something like that, and the canonical absolute dynamic linker file name
for PT_INTERP would be a different one (perhaps just different because of
the different prefix, or perhaps explicitly modified to be ld...-asan).
Then to build applications with this library, you'd point the compiler
driver's paths at the differently prefixed lib directory and pass the
different name to the linker's -dynamic-linker.

Before getting into more details, I think we need to be clear on the vision
of what you want to produce and how it would be used.


Thanks,
Roland
  

On Wed, Aug 27, 2014 at 3:45 PM, Roland McGrath <roland@hack.frob.com> wrote:
> The machinery based on object-suffixes takes care of having separate object
> file builds and archive libraries.  But if you want to make a separate
> flavor of shared objects from those, you'll need to duplicate (refactor)
> a bunch more rules for doing the actual ld -shared step.
>
> I'm inclined to think that this is not really the right model for an ASan
> build.  But to be sure, we need to dig deeper into exactly what you/we
> envision an ASan build containing and how it would be used.
>
> The tack you've attempted leads to the single canonical glibc build
> (optionally) producing another library variant.  The alternate approach is
> a configure switch that changes what the "primary" build does rather than
> adding a variant.  That is, you would configure with --enable-asan (or
> perhaps this should be --with-asan for this approach) as an orthogonal
> switch to --{en,dis}able-{static,shared,profile}.  This build would produce
> a libc.so (and libc.a too unless --disable-static) and ld.so built with
> -fsanitize=address.  You'd configure this for a different --prefix or
> something like that, and the canonical absolute dynamic linker file name
> for PT_INTERP would be a different one (perhaps just different because of
> the different prefix, or perhaps explicitly modified to be ld...-asan).
> Then to build applications with this library, you'd point the compiler
> driver's paths at the differently prefixed lib directory and pass the
> different name to the linker's -dynamic-linker.
>
> Before getting into more details, I think we need to be clear on the vision
> of what you want to produce and how it would be used.

As long as I can link the resulting instrumented libc.so to binaries
and run them
either way works for me. The idea with reusing the -pg approach (which
I think came from Carlos?),
is nice because -pg has similar properties, i.e. we can not add -pg to
objects going to ld.so (right?).
A primary build option (--with-asan) is fine too as long as we can
omit -fsanitize=address when building
some objects (e.g. those for ld.so, maybe a few more).

--kcc
>
>
> Thanks,
> Roland
  

On 08/27/2014 07:08 PM, Konstantin Serebryany wrote:
> On Wed, Aug 27, 2014 at 3:45 PM, Roland McGrath <roland@hack.frob.com> wrote:
>> The machinery based on object-suffixes takes care of having separate object
>> file builds and archive libraries.  But if you want to make a separate
>> flavor of shared objects from those, you'll need to duplicate (refactor)
>> a bunch more rules for doing the actual ld -shared step.
>>
>> I'm inclined to think that this is not really the right model for an ASan
>> build.  But to be sure, we need to dig deeper into exactly what you/we
>> envision an ASan build containing and how it would be used.
>>
>> The tack you've attempted leads to the single canonical glibc build
>> (optionally) producing another library variant.  The alternate approach is
>> a configure switch that changes what the "primary" build does rather than
>> adding a variant.  That is, you would configure with --enable-asan (or
>> perhaps this should be --with-asan for this approach) as an orthogonal
>> switch to --{en,dis}able-{static,shared,profile}.  This build would produce
>> a libc.so (and libc.a too unless --disable-static) and ld.so built with
>> -fsanitize=address.  You'd configure this for a different --prefix or
>> something like that, and the canonical absolute dynamic linker file name
>> for PT_INTERP would be a different one (perhaps just different because of
>> the different prefix, or perhaps explicitly modified to be ld...-asan).
>> Then to build applications with this library, you'd point the compiler
>> driver's paths at the differently prefixed lib directory and pass the
>> different name to the linker's -dynamic-linker.

Configuring with an alternate --prefix defeats one of the main reasons
I want this. I would like to run an entire Fedora bootstrap, or part of
one, with ASan turned on in glibc. Therefore it has to look like dropin
libc.so.6. Nothing in your plan prevents me from doing that, but I just
wanted to point out that alternate --prefix and building everything with
-Wl,--dynamic-linker= goes against my use case. Even though I'm
bootstraping it's always a problem to get flags down into all of the
right places to make the system use a non-default loader (without wrapping
gcc itself).

>> Before getting into more details, I think we need to be clear on the vision
>> of what you want to produce and how it would be used.
> 
> As long as I can link the resulting instrumented libc.so to binaries
> and run them
> either way works for me. The idea with reusing the -pg approach (which
> I think came from Carlos?),
> is nice because -pg has similar properties, i.e. we can not add -pg to
> objects going to ld.so (right?).

Correct, which is why I recommended the -pg approach.

> A primary build option (--with-asan) is fine too as long as we can
> omit -fsanitize=address when building
> some objects (e.g. those for ld.so, maybe a few more).

We can, but then it becomes a set of more custom rules for
building ld.so.

I'm still inclined to think the -pg route is the least work
for ASan, but Roland actually has the best experience to
comment on the amount of work he thinks is required from
each solution.

Cheers,
Carlos.
  

So, Roland, what is your final recommendation?
If you still suggest to use a separate build, do you know if there is
anything similar already in the build system (so that I can duplicate
that)?

Thanks,

--kcc

On Wed, Sep 3, 2014 at 6:41 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 08/27/2014 07:08 PM, Konstantin Serebryany wrote:
>> On Wed, Aug 27, 2014 at 3:45 PM, Roland McGrath <roland@hack.frob.com> wrote:
>>> The machinery based on object-suffixes takes care of having separate object
>>> file builds and archive libraries.  But if you want to make a separate
>>> flavor of shared objects from those, you'll need to duplicate (refactor)
>>> a bunch more rules for doing the actual ld -shared step.
>>>
>>> I'm inclined to think that this is not really the right model for an ASan
>>> build.  But to be sure, we need to dig deeper into exactly what you/we
>>> envision an ASan build containing and how it would be used.
>>>
>>> The tack you've attempted leads to the single canonical glibc build
>>> (optionally) producing another library variant.  The alternate approach is
>>> a configure switch that changes what the "primary" build does rather than
>>> adding a variant.  That is, you would configure with --enable-asan (or
>>> perhaps this should be --with-asan for this approach) as an orthogonal
>>> switch to --{en,dis}able-{static,shared,profile}.  This build would produce
>>> a libc.so (and libc.a too unless --disable-static) and ld.so built with
>>> -fsanitize=address.  You'd configure this for a different --prefix or
>>> something like that, and the canonical absolute dynamic linker file name
>>> for PT_INTERP would be a different one (perhaps just different because of
>>> the different prefix, or perhaps explicitly modified to be ld...-asan).
>>> Then to build applications with this library, you'd point the compiler
>>> driver's paths at the differently prefixed lib directory and pass the
>>> different name to the linker's -dynamic-linker.
>
> Configuring with an alternate --prefix defeats one of the main reasons
> I want this. I would like to run an entire Fedora bootstrap, or part of
> one, with ASan turned on in glibc. Therefore it has to look like dropin
> libc.so.6. Nothing in your plan prevents me from doing that, but I just
> wanted to point out that alternate --prefix and building everything with
> -Wl,--dynamic-linker= goes against my use case. Even though I'm
> bootstraping it's always a problem to get flags down into all of the
> right places to make the system use a non-default loader (without wrapping
> gcc itself).
>
>>> Before getting into more details, I think we need to be clear on the vision
>>> of what you want to produce and how it would be used.
>>
>> As long as I can link the resulting instrumented libc.so to binaries
>> and run them
>> either way works for me. The idea with reusing the -pg approach (which
>> I think came from Carlos?),
>> is nice because -pg has similar properties, i.e. we can not add -pg to
>> objects going to ld.so (right?).
>
> Correct, which is why I recommended the -pg approach.
>
>> A primary build option (--with-asan) is fine too as long as we can
>> omit -fsanitize=address when building
>> some objects (e.g. those for ld.so, maybe a few more).
>
> We can, but then it becomes a set of more custom rules for
> building ld.so.
>
> I'm still inclined to think the -pg route is the least work
> for ASan, but Roland actually has the best experience to
> comment on the amount of work he thinks is required from
> each solution.
>
> Cheers,
> Carlos.
>
  

FTR:
here is the compiler wrapper script that tampers with the compiler
flags to build
ASan-instrumented glibc w/o changing the glibc sources:
https://code.google.com/p/address-sanitizer/source/browse/trunk/asan-glibc/asan-glibc-gcc-wrapper.py

With this I can build the full glibc (tested on 2.19 and 2.20) with
asan instrumentation and find injected bugs.
Now I can move further; however it would still be great if someone can
assist me with properly patching the glibc build system to support
asan build.

--kcc



On Mon, Sep 15, 2014 at 3:11 PM, Konstantin Serebryany
<konstantin.s.serebryany@gmail.com> wrote:
> So, Roland, what is your final recommendation?
> If you still suggest to use a separate build, do you know if there is
> anything similar already in the build system (so that I can duplicate
> that)?
>
> Thanks,
>
> --kcc
>
> On Wed, Sep 3, 2014 at 6:41 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>> On 08/27/2014 07:08 PM, Konstantin Serebryany wrote:
>>> On Wed, Aug 27, 2014 at 3:45 PM, Roland McGrath <roland@hack.frob.com> wrote:
>>>> The machinery based on object-suffixes takes care of having separate object
>>>> file builds and archive libraries.  But if you want to make a separate
>>>> flavor of shared objects from those, you'll need to duplicate (refactor)
>>>> a bunch more rules for doing the actual ld -shared step.
>>>>
>>>> I'm inclined to think that this is not really the right model for an ASan
>>>> build.  But to be sure, we need to dig deeper into exactly what you/we
>>>> envision an ASan build containing and how it would be used.
>>>>
>>>> The tack you've attempted leads to the single canonical glibc build
>>>> (optionally) producing another library variant.  The alternate approach is
>>>> a configure switch that changes what the "primary" build does rather than
>>>> adding a variant.  That is, you would configure with --enable-asan (or
>>>> perhaps this should be --with-asan for this approach) as an orthogonal
>>>> switch to --{en,dis}able-{static,shared,profile}.  This build would produce
>>>> a libc.so (and libc.a too unless --disable-static) and ld.so built with
>>>> -fsanitize=address.  You'd configure this for a different --prefix or
>>>> something like that, and the canonical absolute dynamic linker file name
>>>> for PT_INTERP would be a different one (perhaps just different because of
>>>> the different prefix, or perhaps explicitly modified to be ld...-asan).
>>>> Then to build applications with this library, you'd point the compiler
>>>> driver's paths at the differently prefixed lib directory and pass the
>>>> different name to the linker's -dynamic-linker.
>>
>> Configuring with an alternate --prefix defeats one of the main reasons
>> I want this. I would like to run an entire Fedora bootstrap, or part of
>> one, with ASan turned on in glibc. Therefore it has to look like dropin
>> libc.so.6. Nothing in your plan prevents me from doing that, but I just
>> wanted to point out that alternate --prefix and building everything with
>> -Wl,--dynamic-linker= goes against my use case. Even though I'm
>> bootstraping it's always a problem to get flags down into all of the
>> right places to make the system use a non-default loader (without wrapping
>> gcc itself).
>>
>>>> Before getting into more details, I think we need to be clear on the vision
>>>> of what you want to produce and how it would be used.
>>>
>>> As long as I can link the resulting instrumented libc.so to binaries
>>> and run them
>>> either way works for me. The idea with reusing the -pg approach (which
>>> I think came from Carlos?),
>>> is nice because -pg has similar properties, i.e. we can not add -pg to
>>> objects going to ld.so (right?).
>>
>> Correct, which is why I recommended the -pg approach.
>>
>>> A primary build option (--with-asan) is fine too as long as we can
>>> omit -fsanitize=address when building
>>> some objects (e.g. those for ld.so, maybe a few more).
>>
>> We can, but then it becomes a set of more custom rules for
>> building ld.so.
>>
>> I'm still inclined to think the -pg route is the least work
>> for ASan, but Roland actually has the best experience to
>> comment on the amount of work he thinks is required from
>> each solution.
>>
>> Cheers,
>> Carlos.
>>
  

On 09/23/2014 08:41 PM, Konstantin Serebryany wrote:
> FTR:
> here is the compiler wrapper script that tampers with the compiler
> flags to build
> ASan-instrumented glibc w/o changing the glibc sources:
> https://code.google.com/p/address-sanitizer/source/browse/trunk/asan-glibc/asan-glibc-gcc-wrapper.py
> 
> With this I can build the full glibc (tested on 2.19 and 2.20) with
> asan instrumentation and find injected bugs.
> Now I can move further; however it would still be great if someone can
> assist me with properly patching the glibc build system to support
> asan build.

Have you been able to make progress on logging instead of faulting on
error?

We can help with glibc builds, but it would be nice to make sure that
if we were to turn it on that it won't cause the system to immediately 
abort at boot time when it detects errors.

Recall our conversation at Cauldron 2014, in that ASan + glibc needs to
log errors as precisely as possible for later offline analysis by the
distribution team maintaining glibc.

Does that goal still make sense?

Cheers,
Carlos.
  

On Thu, Sep 25, 2014 at 10:57 AM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 09/23/2014 08:41 PM, Konstantin Serebryany wrote:
>> FTR:
>> here is the compiler wrapper script that tampers with the compiler
>> flags to build
>> ASan-instrumented glibc w/o changing the glibc sources:
>> https://code.google.com/p/address-sanitizer/source/browse/trunk/asan-glibc/asan-glibc-gcc-wrapper.py
>>
>> With this I can build the full glibc (tested on 2.19 and 2.20) with
>> asan instrumentation and find injected bugs.
>> Now I can move further; however it would still be great if someone can
>> assist me with properly patching the glibc build system to support
>> asan build.
>
> Have you been able to make progress on logging instead of faulting on
> error?
>
> We can help with glibc builds, but it would be nice to make sure that
> if we were to turn it on that it won't cause the system to immediately
> abort at boot time when it detects errors.
>
> Recall our conversation at Cauldron 2014, in that ASan + glibc needs to
> log errors as precisely as possible for later offline analysis by the
> distribution team maintaining glibc.
>
> Does that goal still make sense?

Yes, the goal still makes sense.
Today, both GCC and Clang asan implementations have flags to emit
instrumentation via callbacks instead of inline code.
It makes asan somewhat slower, but allows to continue after the first report.

% cat load.c
int load(int *a) { return *a; }
% ~/gcc-inst/bin/gcc -fsanitize=address -O -S -o -   --param
asan-instrumentation-with-call-threshold=0 load.c
pushq %rbx
movq %rdi, %rbx
call __asan_load4
movl (%rbx), %eax
popq %rbx
ret

We may need very minor additional changes in asan-run-time, but we
will need them anyway to support whatever kind of logging you need for
glibc.

--kcc

>
> Cheers,
> Carlos.
>
  

On 09/27/2014 01:21 AM, Konstantin Serebryany wrote:
> Today, both GCC and Clang asan implementations have flags to emit
> instrumentation via callbacks instead of inline code.
> It makes asan somewhat slower, but allows to continue after the first report.

And there is always opportunity to enable logging in inline 
instrumentation as well if outline instrumentation proves to be too slow.

-Y
  

On 03 Sep 2014 21:41, Carlos O'Donell wrote:
> On 08/27/2014 07:08 PM, Konstantin Serebryany wrote:
> > On Wed, Aug 27, 2014 at 3:45 PM, Roland McGrath wrote:
> >> Before getting into more details, I think we need to be clear on the vision
> >> of what you want to produce and how it would be used.
> > 
> > As long as I can link the resulting instrumented libc.so to binaries
> > and run them
> > either way works for me. The idea with reusing the -pg approach (which
> > I think came from Carlos?),
> > is nice because -pg has similar properties, i.e. we can not add -pg to
> > objects going to ld.so (right?).
> 
> Correct, which is why I recommended the -pg approach.

iiuc, the profile option creates a bunch of static libs with _p suffixes right ?  
so instead of -lc, you use -lc_p ?  i don't think that really works for asan.

crazy idea: what about pseudo hwcaps ?  that way you could have asan enabled 
shared libs in an asan/ subdir and the ldso would select them on the fly based 
on your main executable.
-mike