From patchwork Thu Oct 12 09:35:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 23494 Received: (qmail 14962 invoked by alias); 12 Oct 2017 09:35:39 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 14814 invoked by uid 89); 12 Oct 2017 09:35:38 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-25.1 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: EUR03-DB5-obe.outbound.protection.outlook.com From: Wilco Dijkstra To: "libc-alpha@sourceware.org" CC: nd Subject: [PATCH 3/5] Add single-threaded path to _int_free Date: Thu, 12 Oct 2017 09:35:34 +0000 Message-ID: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Wilco.Dijkstra@arm.com; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB6SPR00MB2500; 6:IyNd2MBdauufFTVnpTIm5OWKrysrQEbGHGY0+0D5MCCZ4/tKhFaVPmPZEokPFQZBVb/2r8ZQet0N6VkHIHJoUjD1LKKBusCNuduSNpX0goa3GWVpkXSZFyrvO2fZPLlABCtCH2WlOw43e8Sl8z+RE0+XsZozwGcwx21iMZQgY5eBIroHLCV1uX7+gYy35I72kdubORYUYLSEuzEMeYrc2+P/GONcCi+vPLh2D49KUMKV9BdJPF85PKTkL+Y5X6dGog8I6bc29S+d96vKvcDuF4gXPOT6KIJ92V1i9fZYKDaj0w/bb7R7I4/gkj2/OKtzKrD1l8boOts8uC+Our2Huw==; 5:Mp9TO+4LGAlqA83lu6mixCo/hrhF1d7YY78dQJcsO6nV9540HyUjChlSAhWUfbr9G6dDGmTlpyj1IMtNnL/ik5UZ9g2ioynBg5bIID/d6MXrUBPBj8TGQvgMM0CZEHV1RTMq9AcYbyStfrA8qmmJQw==; 24:4AbrCVajSqpIsjh3t9L9Ixx44LJitc3e4qiM7CitT4bZzyXchiCL1RMChMeJddj64+wkQOOfwnUEoTqOGMMbLX7Pb/r9NPi1yEYldA3ClPg=; 7:f3fIJ6FhXI9cD6FX0aa9qbfMN84x4FBqh4wwKF9CFH1ANxxLHtDHw1YdH/a/Rg4v5BJqhdUgZ2OHOoSbQKpGk3efuLb/aAPgPwgUsAxR6QrMNd/z1BJJfdrofx9st2SVfGEQVcTe+2BY4iyZrlR+vl5n7fQ4/eYxT2wyzo0qPLUjiIFBVvp93u4ua+1VzNxbw0BlBvw/9C78ikfhI+a8KajVQZcz6bUlb6gnZIPfe4E= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 77a3d636-782d-445c-1854-08d511549669 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(48565401081)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:DB6SPR00MB2500; x-ms-traffictypediagnostic: DB6SPR00MB2500: nodisclaimer: True x-exchange-antispam-report-test: UriScan:(180628864354917); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(3002001)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DB6SPR00MB2500; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DB6SPR00MB2500; x-forefront-prvs: 04583CED1A x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(39860400002)(189002)(377424004)(199003)(54534003)(68736007)(189998001)(99286003)(2900100001)(14454004)(2351001)(5660300001)(6116002)(3846002)(102836003)(8676002)(105586002)(106356001)(5640700003)(53936002)(6436002)(2501003)(81156014)(6506006)(8936002)(575784001)(86362001)(3280700002)(3660700001)(81166006)(97736004)(5250100002)(478600001)(101416001)(74316002)(305945005)(66066001)(7736002)(50986999)(6916009)(54356999)(72206003)(33656002)(4326008)(4001150100001)(9686003)(55016002)(25786009)(7696004)(2906002)(316002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6SPR00MB2500; H:DB6PR0801MB2053.eurprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Oct 2017 09:35:34.0406 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6SPR00MB2500 This patch adds single-threaded fast paths to _int_free. Move and simplify the consistency checks so that they don't make the fastbin list update overly complex. Bypass the explicit locking for larger allocations. Passes GLIBC tests, OK for commit? ChangeLog: 2017-10-11 Wilco Dijkstra * malloc/malloc.c (_int_free): Add SINGLE_THREAD_P paths. diff --git a/malloc/malloc.c b/malloc/malloc.c index 13247c64e2be0779050dfbd0fd25f205ba7184f7..c00df205c6004ee5b5d0aee9ffd5130b3c8f9e9f 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -4188,24 +4188,29 @@ _int_free (mstate av, mchunkptr p, int have_lock) /* Atomically link P to its fastbin: P->FD = *FB; *FB = P; */ mchunkptr old = *fb, old2; - unsigned int old_idx = ~0u; - do + + /* Check that the top of the bin is not the record we are going to + add (i.e., double free). */ + if (__builtin_expect (old == p, 0)) + malloc_printerr ("double free or corruption (fasttop)"); + + if (SINGLE_THREAD_P) { - /* Check that the top of the bin is not the record we are going to add - (i.e., double free). */ - if (__builtin_expect (old == p, 0)) - malloc_printerr ("double free or corruption (fasttop)"); - /* Check that size of fastbin chunk at the top is the same as - size of the chunk that we are adding. We can dereference OLD - only if we have the lock, otherwise it might have already been - deallocated. See use of OLD_IDX below for the actual check. */ - if (have_lock && old != NULL) - old_idx = fastbin_index(chunksize(old)); - p->fd = old2 = old; + p->fd = old; + *fb = p; } - while ((old = catomic_compare_and_exchange_val_rel (fb, p, old2)) != old2); - - if (have_lock && old != NULL && __builtin_expect (old_idx != idx, 0)) + else + do + p->fd = old2 = old; + while ((old = catomic_compare_and_exchange_val_rel (fb, p, old2)) + != old2); + + /* Check that size of fastbin chunk at the top is the same as + size of the chunk that we are adding. We can dereference OLD + only if we have the lock, otherwise it might have already been + allocated again. */ + if (have_lock && old != NULL + && __builtin_expect (fastbin_index (chunksize (old)) != idx, 0)) malloc_printerr ("invalid fastbin entry (free)"); } @@ -4214,7 +4219,7 @@ _int_free (mstate av, mchunkptr p, int have_lock) */ else if (!chunk_is_mmapped(p)) { - if (!have_lock) + if (!SINGLE_THREAD_P && !have_lock) __libc_lock_lock (av->mutex); nextchunk = chunk_at_offset(p, size); @@ -4329,7 +4334,7 @@ _int_free (mstate av, mchunkptr p, int have_lock) } } - if (!have_lock) + if (!SINGLE_THREAD_P && !have_lock) __libc_lock_unlock (av->mutex); } /*