From patchwork Fri Sep 6 09:14:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Richard Earnshaw (lists)" X-Patchwork-Id: 34406 Received: (qmail 97612 invoked by alias); 6 Sep 2019 09:14:42 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 97603 invoked by uid 89); 6 Sep 2019 09:14:41 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-18.0 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, SPF_PASS autolearn=ham version=3.3.1 spammy=assist, extreme, granule, 6118 X-HELO: foss.arm.com To: "libc-alpha@sourceware.org" From: "Richard Earnshaw (lists)" Subject: RFD - Support for memory tagging in GLIBC Message-ID: <8306e032-f980-a409-5239-74629e79d041@arm.com> Date: Fri, 6 Sep 2019 10:14:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 NOTE: This code is not yet in a state to be committed, but I wanted to publish something at this point in order to start public discussions on the approach before I develop it further. The code, as published has been tested on a model running a kernel that enables memory tagging, but there are a number of issues that still need to be resolved before I would even consider asking for a merge. I'm not asking for a review of the code as much as a review of the approach at this point in time. I'm posting this now before the Cauldron so hopefully we can have some useful discussions on the matter during that. ARMv8.5 adds an extension known as MTE (memory tagging extension); the extension allows granules of memory (16 bytes per granule) to be individually 'coloured' with a 4-bit tag value. Unused bits in the top byte of a 64-bit address pointer can then be set to describe the colour expected at that address and a protection fault can be raised if there is then a mismatch. This can then be used for a number of purposes, but primarily is intended to assist with debugging a number of run-time faults that are common in code, including buffer overrun faults and use-after-free type errors. Nomenclature: The AArch64 extension is called MTE. I've tried to use the term 'memory tagging' (mtag) in the generic code to keep the layers separate. Ideally mtag can be used on multiple architectures. This patch proposes a way to utilize the extension to provide such protection in Glibc's malloc() API. Essentially the code here is divided into four logical set of changes, though for the purposes of this discussion I've rolled this up into a single patch set. Part 1 is a simple change to the configuration code to allow memory tagging support to built into glibc Part 2 introduces a new (currently internal) API within glibc that provides access to the architecture-specific memory tagging operations. The API essentially compiles down either no-ops or existing standard library functions when the extension is disabled Part 3 is the bulk of the changes to malloc/malloc.c to use the API to colour memory; I've tried to ensure that when the extension is disabled there is no overhead on existing users. If the extension is enabled during the build, but disabled at run time (eg to support systems that do not have the extension), then there are some minor overheads, but they are hopefully not significant. Part 4 is finally some target specific changes for AArch64; when MTE is enabled we have to be very careful about buffer overruns on read operations. Consequently we have to constrain some of the string operations to ensure that they do not unsafely read across a tag granule boundary. This code is very preliminary at present - eventually we would want to be able to select the code at run time and revert back to the standard code if tagging is disabled. Parts 2 and 3 are obviously the focus of the discussion I'd like to have at present. For part 2, the API is currently entirely private within glibc, but potentially in future (once we're happy that the concept is stable) it might be useful to open this up to users as well. Part 3, which is the bulk of the changes colours all memory requests obtained through the malloc API. Each call to malloc (or any of the more aligned variants) or realloc will return a coloured pointer if the extension is enabled - for realloc I've chosen to recolour all affected memory even if the same logical address is returned (the pointer will contain a different colour, ensuring that before-and-after pointers will not compare equal). This is perhaps the most extreme position, but in some cases that might catch assumptions or code that continues to use the pre-realloc address incorrectly. - colouring is mostly done at the outermost level of the library. This is not necessarily the most efficient point to do this, but it certainly creates the least disruption in the code base. The main exception to this is realloc where the separation of the layers is not quite as clean as for the other entry points. The advantage of colouring at the outermost level is that calloc() can combine the clearing of the memory with the colouring process if the architecture supports that (MTE does). - one colour is retained for use by libc itself. This can be used to detect when user code goes outside the allocated buffer region. - free() recolours the memory; this is a run-time overhead but is useful for catching use-after-free accesses. Limitations in the prototype: MTE has a granule size (minimum colourable memory block) of 16 bytes. This happens to fit well with malloc's block header, which on aarch64 is also 16 bytes of size and thus leads to little overhead in the data structures. I haven't attempted yet to look at support for other sizes, but I suspect that things will become a bit messy if the granule is larger than the block header (it will certainly be less efficient). At present, the code simply assumes that all memory could be tagged (though it works correctly if it is not). We are in discussions with the kernel folk about the possible syscall API extensions that might be needed to make requests for memory from the kernel tagable. I've written enough for now. Let the discussions begin... ----- [mtag] Allow memory tagging to be enabled from the command line This patch adds the configuration machinery to allow memory tagging to be enabled from the command line via the configure option --enable-memory-tagging. The current default is off, though in time we may change that once the API is more stable. [AArch64][mtag] Basic support for memory tagging This patch adds the basic support for memory tagging. This is very much preliminary code and is unlikely to be in its final form. - generic/libc-mtag.h - default implementation of the memory tagging interface. Maps most functions onto no-ops, a few are mapped back onto existing APIs (eg memset). - aarch64/libc-mtag.h - implementation for AArch64. - aarch64/__mtag_* - helper functions for memory tagging (unoptimized). - malloc/malloc.c - updates to support tagging of memory allocations. [AArch64][mtag] Mitigations for string functions when MTE is enabled. This is an initial set of patches for mitigating against MTE issues when that is enabled. Most of the changes are sub-optimal, but should avoid the boundary conditions that can cause spurious MTE faults. diff --git a/Makeconfig b/Makeconfig index 0e386fbc19..fb5ffa84be 100644 --- a/Makeconfig +++ b/Makeconfig @@ -1327,6 +1327,11 @@ defines += -D_LIBC_REENTRANT libio-mtsafe = -D_IO_MTSAFE_IO endif +# Enable memory tagging, if supported by the architecture +ifeq (yes,$(memory-tagging)) +defines += -D_LIBC_MTAG +endif + # The name to give to a test in test results summaries. test-name = $(strip $(patsubst %.out, %, $(patsubst $(common-objpfx)%, %, $@))) diff --git a/config.make.in b/config.make.in index 2fed3da773..b3be8e0381 100644 --- a/config.make.in +++ b/config.make.in @@ -84,6 +84,8 @@ mach-interface-list = @mach_interface_list@ experimental-malloc = @experimental_malloc@ +memory-tagging = @memory_tagging@ + nss-crypt = @libc_cv_nss_crypt@ static-nss-crypt = @libc_cv_static_nss_crypt@ diff --git a/configure b/configure index c773c487b5..8e2dd7835d 100755 --- a/configure +++ b/configure @@ -678,6 +678,7 @@ link_obsolete_rpc libc_cv_static_nss_crypt libc_cv_nss_crypt build_crypt +memory_tagging experimental_malloc enable_werror all_warnings @@ -783,6 +784,7 @@ enable_all_warnings enable_werror enable_multi_arch enable_experimental_malloc +enable_memory_tagging enable_crypt enable_nss_crypt enable_obsolete_rpc @@ -1454,6 +1456,8 @@ Optional Features: architectures --disable-experimental-malloc disable experimental malloc features + --enable-memory-tagging enable memory tagging if supported by the + architecture --disable-crypt do not build nor install the passphrase hashing library, libcrypt --enable-nss-crypt enable libcrypt to use nss @@ -3527,6 +3531,15 @@ fi +# Check whether --enable-memory-tagging was given. +if test "${enable_memory_tagging+set}" = set; then : + enableval=$enable_memory_tagging; memory_tagging=$enableval +else + memory_tagging=no +fi + + + # Check whether --enable-crypt was given. if test "${enable_crypt+set}" = set; then : enableval=$enable_crypt; build_crypt=$enableval diff --git a/configure.ac b/configure.ac index 598ba6c4ae..69708cd924 100644 --- a/configure.ac +++ b/configure.ac @@ -310,6 +310,13 @@ AC_ARG_ENABLE([experimental-malloc], [experimental_malloc=yes]) AC_SUBST(experimental_malloc) +AC_ARG_ENABLE([memory-tagging], + AC_HELP_STRING([--enable-memory-tagging], + [enable memory tagging if supported by the architecture]), + [memory_tagging=$enableval], + [memory_tagging=no]) +AC_SUBST(memory_tagging) + AC_ARG_ENABLE([crypt], AC_HELP_STRING([--disable-crypt], [do not build nor install the passphrase hashing library, libcrypt]), diff --git a/malloc/malloc.c b/malloc/malloc.c index 00ce48cf58..af59badf8f 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -242,6 +242,9 @@ /* For DIAG_PUSH/POP_NEEDS_COMMENT et al. */ #include +/* For memory tagging. */ +#include + #include /* For SINGLE_THREAD_P. */ @@ -279,6 +282,13 @@ #define MALLOC_DEBUG 0 #endif +/* When using tagged memory, we cannot share the end of the user block + with the header for the next chunk, so ensure that we allocate + blocks that are rounded up to the granue size. */ +#define ROUND_UP_ALLOCATION_SIZE(BYTES) \ + ((size_t) (((INTERNAL_SIZE_T)(BYTES) + __MTAG_GRANULE_SIZE - 1) \ + & ~(__MTAG_GRANULE_SIZE - 1))) + #ifndef NDEBUG # define __assert_fail(assertion, file, line, function) \ __malloc_assert(assertion, file, line, function) @@ -1173,7 +1183,8 @@ nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ /* conversion from malloc headers to user pointers, and back */ #define chunk2mem(p) ((void*)((char*)(p) + 2*SIZE_SZ)) -#define mem2chunk(mem) ((mchunkptr)((char*)(mem) - 2*SIZE_SZ)) +#define mem2chunk(mem) \ + ((mchunkptr)__libc_mtag_address_get_tag (((char*)(mem) - 2*SIZE_SZ))) /* The smallest possible chunk */ #define MIN_CHUNK_SIZE (offsetof(struct malloc_chunk, fd_nextsize)) @@ -3031,6 +3042,7 @@ __libc_malloc (size_t bytes) = atomic_forced_read (__malloc_hook); if (__builtin_expect (hook != NULL, 0)) return (*hook)(bytes, RETURN_ADDRESS (0)); + bytes = ROUND_UP_ALLOCATION_SIZE (bytes); #if USE_TCACHE /* int_free also calls request2size, be careful to not pad twice. */ size_t tbytes; @@ -3048,7 +3060,9 @@ __libc_malloc (size_t bytes) && tcache && tcache->counts[tc_idx] > 0) { - return tcache_get (tc_idx); + victim = tcache_get (tc_idx); + return __libc_mtag_tag_region (__libc_mtag_new_tag (victim), + __malloc_usable_size (victim)); } DIAG_POP_NEEDS_COMMENT; #endif @@ -3056,6 +3070,9 @@ __libc_malloc (size_t bytes) if (SINGLE_THREAD_P) { victim = _int_malloc (&main_arena, bytes); + if (victim) + victim = __libc_mtag_tag_region (__libc_mtag_new_tag (victim), + __malloc_usable_size (victim)); assert (!victim || chunk_is_mmapped (mem2chunk (victim)) || &main_arena == arena_for_chunk (mem2chunk (victim))); return victim; @@ -3076,6 +3093,10 @@ __libc_malloc (size_t bytes) if (ar_ptr != NULL) __libc_lock_unlock (ar_ptr->mutex); + if (victim) + victim = __libc_mtag_tag_region (__libc_mtag_new_tag (victim), + __malloc_usable_size (victim)); + assert (!victim || chunk_is_mmapped (mem2chunk (victim)) || ar_ptr == arena_for_chunk (mem2chunk (victim))); return victim; @@ -3099,8 +3120,17 @@ __libc_free (void *mem) if (mem == 0) /* free(0) has no effect */ return; +#ifdef _LIBC_MTAG + /* Quickly check that the freed pointer matches the tag for the memory. + This gives a useful double-free detection. */ + *(volatile char *)mem; +#endif + p = mem2chunk (mem); + /* Mark the chunk as belonging to the library again. */ + (void)__libc_mtag_tag_region (chunk2mem (p), __malloc_usable_size (mem)); + if (chunk_is_mmapped (p)) /* release mmapped memory. */ { /* See if the dynamic brk/mmap threshold needs adjusting. @@ -3150,6 +3180,13 @@ __libc_realloc (void *oldmem, size_t bytes) if (oldmem == 0) return __libc_malloc (bytes); + bytes = ROUND_UP_ALLOCATION_SIZE (bytes); +#ifdef _LIBC_MTAG + /* Perform a quick check to ensure that the pointer's tag matches the + memory's tag. */ + *(volatile char*) oldmem; +#endif + /* chunk corresponding to oldmem */ const mchunkptr oldp = mem2chunk (oldmem); /* its size */ @@ -3205,7 +3242,17 @@ __libc_realloc (void *oldmem, size_t bytes) #if HAVE_MREMAP newp = mremap_chunk (oldp, nb); if (newp) - return chunk2mem (newp); + { + void *newmem = chunk2mem (newp); + /* Give the new block a different tag. This helps to ensure + that stale handles to the previous mapping are not + reused. There's a performance hit for both us and the + caller for doing this, so we might want to + reconsider. */ + newmem = __libc_mtag_tag_region (__libc_mtag_new_tag (newmem), + __malloc_usable_size (newmem)); + return newmem; + } #endif /* Note the extra SIZE_SZ overhead. */ if (oldsize - SIZE_SZ >= nb) @@ -3288,7 +3335,7 @@ _mid_memalign (size_t alignment, size_t bytes, void *address) return 0; } - + bytes = ROUND_UP_ALLOCATION_SIZE (bytes); /* Make sure alignment is power of 2. */ if (!powerof2 (alignment)) { @@ -3303,8 +3350,8 @@ _mid_memalign (size_t alignment, size_t bytes, void *address) p = _int_memalign (&main_arena, alignment, bytes); assert (!p || chunk_is_mmapped (mem2chunk (p)) || &main_arena == arena_for_chunk (mem2chunk (p))); - - return p; + return p ? __libc_mtag_tag_region (__libc_mtag_new_tag (p), + __malloc_usable_size (p)) : NULL; } arena_get (ar_ptr, bytes + alignment + MINSIZE); @@ -3322,6 +3369,8 @@ _mid_memalign (size_t alignment, size_t bytes, void *address) assert (!p || chunk_is_mmapped (mem2chunk (p)) || ar_ptr == arena_for_chunk (mem2chunk (p))); + return p ? __libc_mtag_tag_region (__libc_mtag_new_tag (p), + __malloc_usable_size (p)) : NULL; return p; } /* For ISO C11. */ @@ -3331,20 +3380,28 @@ libc_hidden_def (__libc_memalign) void * __libc_valloc (size_t bytes) { + void *p; + if (__malloc_initialized < 0) ptmalloc_init (); + bytes = ROUND_UP_ALLOCATION_SIZE (bytes); void *address = RETURN_ADDRESS (0); size_t pagesize = GLRO (dl_pagesize); - return _mid_memalign (pagesize, bytes, address); + p = _mid_memalign (pagesize, bytes, address); + return p ? __libc_mtag_tag_region (__libc_mtag_new_tag (p), + __malloc_usable_size (p)) : NULL; } void * __libc_pvalloc (size_t bytes) { + void *p; + if (__malloc_initialized < 0) ptmalloc_init (); + bytes = ROUND_UP_ALLOCATION_SIZE (bytes); void *address = RETURN_ADDRESS (0); size_t pagesize = GLRO (dl_pagesize); size_t rounded_bytes; @@ -3358,7 +3415,9 @@ __libc_pvalloc (size_t bytes) } rounded_bytes = rounded_bytes & -(pagesize - 1); - return _mid_memalign (pagesize, rounded_bytes, address); + p = _mid_memalign (pagesize, rounded_bytes, address); + return p ? __libc_mtag_tag_region (__libc_mtag_new_tag (p), + __malloc_usable_size (p)) : NULL; } void * @@ -3368,9 +3427,11 @@ __libc_calloc (size_t n, size_t elem_size) mchunkptr oldtop, p; INTERNAL_SIZE_T sz, csz, oldtopsize; void *mem; +#ifndef _LIBC_MTAG unsigned long clearsize; unsigned long nclears; INTERNAL_SIZE_T *d; +#endif ptrdiff_t bytes; if (__glibc_unlikely (__builtin_mul_overflow (n, elem_size, &bytes))) @@ -3378,6 +3439,7 @@ __libc_calloc (size_t n, size_t elem_size) __set_errno (ENOMEM); return NULL; } + sz = bytes; void *(*hook) (size_t, const void *) = @@ -3393,6 +3455,7 @@ __libc_calloc (size_t n, size_t elem_size) MAYBE_INIT_TCACHE (); + sz = ROUND_UP_ALLOCATION_SIZE (sz); if (SINGLE_THREAD_P) av = &main_arena; else @@ -3447,7 +3510,15 @@ __libc_calloc (size_t n, size_t elem_size) if (mem == 0) return 0; + /* If we are using memory tagging, then we need to set the tags + regardless of MORECORE_CLEARS, so we zero the whole block while + doing so. */ +#ifdef _LIBC_MTAG + return __libc_mtag_memset_with_tag (__libc_mtag_new_tag (mem), 0, + __malloc_usable_size (mem)); +#else p = mem2chunk (mem); + csz = chunksize (p); /* Two optional cases in which clearing not necessary */ if (chunk_is_mmapped (p)) @@ -3458,8 +3529,6 @@ __libc_calloc (size_t n, size_t elem_size) return mem; } - csz = chunksize (p); - #if MORECORE_CLEARS if (perturb_byte == 0 && (p == oldtop && csz > oldtopsize)) { @@ -3502,6 +3571,7 @@ __libc_calloc (size_t n, size_t elem_size) } return mem; +#endif } /* @@ -4581,7 +4651,9 @@ _int_realloc(mstate av, mchunkptr oldp, INTERNAL_SIZE_T oldsize, av->top = chunk_at_offset (oldp, nb); set_head (av->top, (newsize - nb) | PREV_INUSE); check_inuse_chunk (av, oldp); - return chunk2mem (oldp); + newmem = chunk2mem (oldp); + return __libc_mtag_tag_region (__libc_mtag_new_tag (newmem), + __malloc_usable_size (newmem)); } /* Try to expand forward into next chunk; split off remainder below */ @@ -4614,7 +4686,10 @@ _int_realloc(mstate av, mchunkptr oldp, INTERNAL_SIZE_T oldsize, } else { + newmem = __libc_mtag_tag_region (__libc_mtag_new_tag (newmem), + __malloc_usable_size (newmem)); memcpy (newmem, chunk2mem (oldp), oldsize - SIZE_SZ); + (void) __libc_mtag_tag_region (chunk2mem (oldp), oldsize); _int_free (av, oldp, 1); check_inuse_chunk (av, newp); return chunk2mem (newp); @@ -4636,6 +4711,8 @@ _int_realloc(mstate av, mchunkptr oldp, INTERNAL_SIZE_T oldsize, else /* split remainder */ { remainder = chunk_at_offset (newp, nb); + /* Clear any user-space tags before writing the header. */ + remainder = __libc_mtag_tag_region (remainder, remainder_size); set_head_size (newp, nb | (av != &main_arena ? NON_MAIN_ARENA : 0)); set_head (remainder, remainder_size | PREV_INUSE | (av != &main_arena ? NON_MAIN_ARENA : 0)); @@ -4645,7 +4722,9 @@ _int_realloc(mstate av, mchunkptr oldp, INTERNAL_SIZE_T oldsize, } check_inuse_chunk (av, newp); - return chunk2mem (newp); + newmem = chunk2mem (newp); + return __libc_mtag_tag_region (__libc_mtag_new_tag (newmem), + __malloc_usable_size (newmem)); } /* @@ -4865,7 +4944,7 @@ __malloc_usable_size (void *m) size_t result; result = musable (m); - return result; + return (size_t) (((INTERNAL_SIZE_T)result) & ~(__MTAG_GRANULE_SIZE - 1)); } /* diff --git a/sysdeps/aarch64/Makefile b/sysdeps/aarch64/Makefile index 94baaf52dd..9f8e90da23 100644 --- a/sysdeps/aarch64/Makefile +++ b/sysdeps/aarch64/Makefile @@ -16,3 +16,11 @@ endif ifeq ($(subdir),math) CPPFLAGS += -I../soft-fp endif + +ifeq ($(subdir),string) +sysdep_routines += __mtag_memset_tag +endif + +ifeq ($(subdir),misc) +sysdep_routines += __mtag_tag_region __mtag_new_tag __mtag_address_get_tag +endif diff --git a/sysdeps/aarch64/__mtag_address_get_tag.S b/sysdeps/aarch64/__mtag_address_get_tag.S new file mode 100644 index 0000000000..681950e92e --- /dev/null +++ b/sysdeps/aarch64/__mtag_address_get_tag.S @@ -0,0 +1,31 @@ +/* Copyright (C) 2019 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + +#define ptr x0 + + .arch armv8.5-a + .arch_extension memtag + +ENTRY (__libc_mtag_address_get_tag) + + ldg ptr, [ptr] + ret +END (__libc_mtag_address_get_tag) +libc_hidden_builtin_def (__libc_mtag_address_get_tag) diff --git a/sysdeps/aarch64/__mtag_memset_tag.S b/sysdeps/aarch64/__mtag_memset_tag.S new file mode 100644 index 0000000000..c3f424b57b --- /dev/null +++ b/sysdeps/aarch64/__mtag_memset_tag.S @@ -0,0 +1,46 @@ +/* Copyright (C) 2019 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +/* Use the same register names and assignments as memset. */ +#include "memset-reg.h" + + .arch armv8.5-a + .arch_extension memtag + +/* NB, only supported on variants with 64-bit pointers. */ + +/* FIXME: This is a minimal implementation. We could do much better than + this for large values of COUNT. */ + +ENTRY_ALIGN(__libc_mtag_memset_with_tag, 6) + + and valw, valw, 255 + orr valw, valw, valw, lsl 8 + orr valw, valw, valw, lsl 16 + orr val, val, val, lsl 32 + mov dst, dstin + +L(loop): + stgp val, val, [dst], #16 + subs count, count, 16 + bne L(loop) + ldg dstin, [dstin] // Recover the tag created (might be untagged). + ret +END (__libc_mtag_memset_with_tag) +libc_hidden_builtin_def (__libc_mtag_memset_with_tag) diff --git a/sysdeps/aarch64/__mtag_new_tag.S b/sysdeps/aarch64/__mtag_new_tag.S new file mode 100644 index 0000000000..45bffc97b7 --- /dev/null +++ b/sysdeps/aarch64/__mtag_new_tag.S @@ -0,0 +1,38 @@ +/* Copyright (C) 2019 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + + .arch armv8.5-a + .arch_extension memtag + +/* NB, only supported on variants with 64-bit pointers. */ + +/* FIXME: This is a minimal implementation. We could do better than + this for larger values of COUNT. */ + +#define ptr x0 +#define xset x1 + +ENTRY(__libc_mtag_new_tag) + // Guarantee that the new tag is not the same as now. + gmi xset, ptr, xzr + irg ptr, ptr, xset + ret +END (__libc_mtag_new_tag) +libc_hidden_builtin_def (__libc_mtag_new_tag) diff --git a/sysdeps/aarch64/__mtag_tag_region.S b/sysdeps/aarch64/__mtag_tag_region.S new file mode 100644 index 0000000000..50d03a9097 --- /dev/null +++ b/sysdeps/aarch64/__mtag_tag_region.S @@ -0,0 +1,44 @@ +/* Copyright (C) 2019 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +/* Use the same register names and assignments as memset. */ + + .arch armv8.5-a + .arch_extension memtag + +/* NB, only supported on variants with 64-bit pointers. */ + +/* FIXME: This is a minimal implementation. We could do better than + this for larger values of COUNT. */ + +#define dstin x0 +#define count x1 +#define dst x2 + +ENTRY_ALIGN(__libc_mtag_tag_region, 6) + + mov dst, dstin +L(loop): + stg dst, [dst], #16 + subs count, count, 16 + bne L(loop) + ldg dstin, [dstin] // Recover the tag created (might be untagged). + ret +END (__libc_mtag_tag_region) +libc_hidden_builtin_def (__libc_mtag_tag_region) diff --git a/sysdeps/aarch64/libc-mtag.h b/sysdeps/aarch64/libc-mtag.h new file mode 100644 index 0000000000..b9cf09d561 --- /dev/null +++ b/sysdeps/aarch64/libc-mtag.h @@ -0,0 +1,54 @@ +/* libc-internal interface for tagged (colored) memory support. + Copyright (C) 2019 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _AARCH64_LIBC_MTAG_H +#define _AARCH64_LIBC_MTAG_H 1 + +#ifndef _LIBC_MTAG +/* Generic bindings for systems that do not support memory tagging. */ +#include_next "libc-mtag.h" +#else + +/* Used to ensure additional alignment when objects need to have distinct + tags. */ +#define __MTAG_GRANULE_SIZE 16 + +/* Non-zero if memory obtained via morecore (sbrk) is not tagged. */ +#define __MTAG_SBRK_UNTAGGED 0 + +/* Set the tags for a region of memory, which must have size and alignment + that are multiples of __MTAG_GRANULE_SIZE. Size cannot be zero. + void *__libc_mtag_tag_region (const void *, size_t) */ +void *__libc_mtag_tag_region (void *, size_t); + +/* Optimized equivalent to __libc_mtag_tag_region followed by memset. */ +void *__libc_mtag_memset_with_tag(void *, int, size_t); + +/* Convert address P to a pointer that is tagged correctly for that + location. + void *__libc_mtag_address_get_tag (void*) */ +void *__libc_mtag_address_get_tag(void *); + +/* Assign a new (random) tag to a pointer P (does not adjust the tag on + the memory addressed). + void *__libc_mtag_new_tag (void*) */ +void *__libc_mtag_new_tag(void *); + +#endif /* _LIBC_MTAG */ + +#endif /* _AARCH64_LIBC_MTAG_H */ diff --git a/sysdeps/aarch64/memchr.S b/sysdeps/aarch64/memchr.S index 7023d913e7..b766ed4b79 100644 --- a/sysdeps/aarch64/memchr.S +++ b/sysdeps/aarch64/memchr.S @@ -64,6 +64,25 @@ */ ENTRY (MEMCHR) +#ifdef _LIBC_MTAG + /* Quick-and-dirty implementation for MTE. Needs a rewrite as + granules are only 16 bytes in size. */ + /* Do not dereference srcin if no bytes to compare. */ + cbz cntin, L(zero_length) + and chrin, chrin, #255 +L(next_byte): + ldrb wtmp2, [srcin], #1 + cmp wtmp2, chrin + b.eq L(found) + subs cntin, cntin, #1 + b.ne L(next_byte) +L(zero_length): + mov result, #0 + ret +L(found): + sub result, srcin, #1 + ret +#else /* Do not dereference srcin if no bytes to compare. */ cbz cntin, L(zero_length) /* @@ -152,10 +171,10 @@ L(tail): /* Select result or NULL */ csel result, xzr, result, eq ret - L(zero_length): mov result, #0 ret +#endif /* _LIBC_MTAG */ END (MEMCHR) weak_alias (MEMCHR, memchr) libc_hidden_builtin_def (memchr) diff --git a/sysdeps/aarch64/multiarch/strlen_asimd.S b/sysdeps/aarch64/multiarch/strlen_asimd.S index 9316c7481b..010309bee5 100644 --- a/sysdeps/aarch64/multiarch/strlen_asimd.S +++ b/sysdeps/aarch64/multiarch/strlen_asimd.S @@ -48,7 +48,7 @@ #define dataq2 q3 #define datav2 v3 -#ifdef TEST_PAGE_CROSS +#if defined _LIBC_MTAG || defined TEST_PAGE_CROSS # define MIN_PAGE_SIZE 16 #else # define MIN_PAGE_SIZE 4096 diff --git a/sysdeps/aarch64/strchr.S b/sysdeps/aarch64/strchr.S index 770bcd20df..66f1b1ce11 100644 --- a/sysdeps/aarch64/strchr.S +++ b/sysdeps/aarch64/strchr.S @@ -63,6 +63,20 @@ ENTRY (strchr) DELOUSE (0) +#ifdef _LIBC_MTAG + /* Quick and dirty implementation for MTE */ + and chrin, chrin, #255 +L(next_byte): + ldrb wtmp2, [srcin], #1 + cbz wtmp2, L(end) + cmp wtmp2, chrin + b.ne L(next_byte) + sub result, srcin, #1 + ret +L(end): + mov result, #0 + ret +#else mov wtmp2, #0x0401 movk wtmp2, #0x4010, lsl #16 dup vrepchr.16b, chrin @@ -134,6 +148,7 @@ L(tail): add result, src, tmp1, lsr #1 csel result, result, xzr, eq ret +#endif END (strchr) libc_hidden_builtin_def (strchr) weak_alias (strchr, index) diff --git a/sysdeps/aarch64/strchrnul.S b/sysdeps/aarch64/strchrnul.S index 8ddb516d18..c63b6c6285 100644 --- a/sysdeps/aarch64/strchrnul.S +++ b/sysdeps/aarch64/strchrnul.S @@ -61,6 +61,18 @@ ENTRY (__strchrnul) DELOUSE (0) +#ifdef _LIBC_MTAG + /* Quick and dirty implementation for MTE */ + and chrin, chrin, #255 +L(next_byte): + ldrb wtmp2, [srcin], #1 + cmp wtmp2, #0 + ccmp wtmp2, chrin, #4, ne /* NZCV = 0x0100 */ + b.ne L(next_byte) + + sub result, srcin, #1 + ret +#else /* Magic constant 0x40100401 to allow us to identify which lane matches the termination condition. */ mov wtmp2, #0x0401 @@ -126,6 +138,6 @@ L(tail): /* tmp1 is twice the offset into the fragment. */ add result, src, tmp1, lsr #1 ret - +#endif /* _LIBC_MTAG */ END(__strchrnul) weak_alias (__strchrnul, strchrnul) diff --git a/sysdeps/aarch64/strcmp.S b/sysdeps/aarch64/strcmp.S index f63a6c90a8..1926769dc4 100644 --- a/sysdeps/aarch64/strcmp.S +++ b/sysdeps/aarch64/strcmp.S @@ -46,6 +46,12 @@ #define zeroones x10 #define pos x11 +#if defined _LIBC_MTAG || defined TEST_PAGE_CROSS +# define MIN_PAGE_SIZE 16 +#else +# define MIN_PAGE_SIZE 4096 +#endif + /* Start of performance-critical section -- one 64B cache line. */ ENTRY_ALIGN(strcmp, 6) @@ -161,10 +167,10 @@ L(do_misaligned): b.ne L(do_misaligned) L(loop_misaligned): - /* Test if we are within the last dword of the end of a 4K page. If + /* Test if we are within the last dword of the end of a page. If yes then jump back to the misaligned loop to copy a byte at a time. */ - and tmp1, src2, #0xff8 - eor tmp1, tmp1, #0xff8 + and tmp1, src2, #(MIN_PAGE_SIZE - 8) + eor tmp1, tmp1, #(MIN_PAGE_SIZE - 8) cbz tmp1, L(do_misaligned) ldr data1, [src1], #8 ldr data2, [src2], #8 diff --git a/sysdeps/aarch64/strcpy.S b/sysdeps/aarch64/strcpy.S index f267eafa8d..bc7b7aae4d 100644 --- a/sysdeps/aarch64/strcpy.S +++ b/sysdeps/aarch64/strcpy.S @@ -81,7 +81,7 @@ misaligned, crosses a page boundary - after that we move to aligned fetches for the remainder of the string. */ -#ifdef STRCPY_TEST_PAGE_CROSS +#if defined _LIBC_MTAG || defined STRCPY_TEST_PAGE_CROSS /* Make everything that isn't Qword aligned look like a page cross. */ #define MIN_PAGE_P2 4 #else diff --git a/sysdeps/aarch64/strlen.S b/sysdeps/aarch64/strlen.S index cf5e7a9362..5c3c0bcbf2 100644 --- a/sysdeps/aarch64/strlen.S +++ b/sysdeps/aarch64/strlen.S @@ -57,7 +57,7 @@ #define REP8_7f 0x7f7f7f7f7f7f7f7f #define REP8_80 0x8080808080808080 -#ifdef TEST_PAGE_CROSS +#if defined _LIBC_MTAG || defined TEST_PAGE_CROSS # define MIN_PAGE_SIZE 16 #else # define MIN_PAGE_SIZE 4096 diff --git a/sysdeps/aarch64/strncmp.S b/sysdeps/aarch64/strncmp.S index 00cc627aca..1e9c221f22 100644 --- a/sysdeps/aarch64/strncmp.S +++ b/sysdeps/aarch64/strncmp.S @@ -51,6 +51,12 @@ #define endloop x15 #define count mask +#if defined _LIBC_MTAG || defined TEST_PAGE_CROSS +# define MIN_PAGE_SIZE 16 +#else +# define MIN_PAGE_SIZE 4096 +#endif + ENTRY_ALIGN_AND_PAD (strncmp, 6, 7) DELOUSE (0) DELOUSE (1) @@ -233,8 +239,8 @@ L(do_misaligned): subs limit_wd, limit_wd, #1 b.lo L(done_loop) L(loop_misaligned): - and tmp2, src2, #0xff8 - eor tmp2, tmp2, #0xff8 + and tmp2, src2, #(MIN_PAGE_SIZE - 8) + eor tmp2, tmp2, #(MIN_PAGE_SIZE - 8) cbz tmp2, L(page_end_loop) ldr data1, [src1], #8 diff --git a/sysdeps/aarch64/strrchr.S b/sysdeps/aarch64/strrchr.S index c064951c73..5d8daadd1f 100644 --- a/sysdeps/aarch64/strrchr.S +++ b/sysdeps/aarch64/strrchr.S @@ -70,6 +70,19 @@ ENTRY(strrchr) DELOUSE (0) cbz x1, L(null_search) +#ifdef _LIBC_MTAG + /* Quick and dirty version for MTE. */ + and chrin, chrin, #255 + mov src_match, #0 +L(next_byte): + ldrb wtmp2, [srcin] + cmp wtmp2, chrin + csel src_match, src_match, srcin, ne + add srcin, srcin, #1 + cbnz wtmp2, L(next_byte) + mov result, src_match + ret +#else /* Magic constant 0x40100401 to allow us to identify which lane matches the requested byte. Magic constant 0x80200802 used similarly for NUL termination. */ @@ -158,9 +171,9 @@ L(tail): csel result, result, xzr, ne ret +#endif L(null_search): b __strchrnul - END(strrchr) weak_alias (strrchr, rindex) libc_hidden_builtin_def (strrchr) diff --git a/sysdeps/generic/libc-mtag.h b/sysdeps/generic/libc-mtag.h new file mode 100644 index 0000000000..76bf328039 --- /dev/null +++ b/sysdeps/generic/libc-mtag.h @@ -0,0 +1,49 @@ +/* libc-internal interface for tagged (colored) memory support. + Copyright (C) 2019 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _GENERIC_LIBC_MTAG_H +#define _GENERIC_LIBC_MTAG_H 1 + +/* Generic bindings for systems that do not support memory tagging. */ + +/* Used to ensure additional alignment when objects need to have distinct + tags. */ +#define __MTAG_GRANULE_SIZE 1 + +/* Non-zero if memory obtained via morecore (sbrk) is not tagged. */ +#define __MTAG_SBRK_UNTAGGED 0 + +/* Set the tags for a region of memory, which must have size and alignment + that are multiples of __MTAG_GRANULE_SIZE. Size cannot be zero. + void *__libc_mtag_tag_region (const void *, size_t) */ +#define __libc_mtag_tag_region(p, s) (p) + +/* Optimized equivalent to __libc_mtag_tag_region followed by memset. */ +#define __libc_mtag_memset_with_tag memset + +/* Convert address P to a pointer that is tagged correctly for that + location. + void *__libc_mtag_address_get_tag (void*) */ +#define __libc_mtag_address_get_tag(p) (p) + +/* Assign a new (random) tag to a pointer P (does not adjust the tag on + the memory addressed). + void *__libc_mtag_new_tag (void*) */ +#define __libc_mtag_new_tag(p) (p) + +#endif /* _GENERIC_LIBC_MTAG_H */