From patchwork Fri Jun 8 17:25:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gary Benson X-Patchwork-Id: 27720 Received: (qmail 44700 invoked by alias); 8 Jun 2018 17:25:49 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Delivered-To: mailing list gdb-patches@sourceware.org Received: (qmail 44678 invoked by uid 89); 8 Jun 2018 17:25:48 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-23.2 required=5.0 tests=AWL, BAYES_00, DIET_1, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=XML, descriptions, Prints X-HELO: mx1.redhat.com Received: from mx3-rdu2.redhat.com (HELO mx1.redhat.com) (66.187.233.73) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 08 Jun 2018 17:25:45 +0000 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D8DCD103B77; Fri, 8 Jun 2018 17:25:43 +0000 (UTC) Received: from blade.nx (ovpn-116-229.ams2.redhat.com [10.36.116.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id 49429ED16A; Fri, 8 Jun 2018 17:25:43 +0000 (UTC) Received: from blade.com (localhost [127.0.0.1]) by blade.nx (Postfix) with ESMTP id 7456C816CD7C; Fri, 8 Jun 2018 18:25:42 +0100 (BST) From: Gary Benson To: gdb-patches@sourceware.org Cc: Pedro Alves , Simon Marchi , Eli Zaretskii Subject: [PATCH v3/pushed] linux: Add maintenance commands to test libthread_db Date: Fri, 8 Jun 2018 18:25:39 +0100 Message-Id: <1528478739-23579-1-git-send-email-gbenson@redhat.com> In-Reply-To: <20074994-70c2-151c-95c3-5c0090a87bd8@redhat.com> References: <20074994-70c2-151c-95c3-5c0090a87bd8@redhat.com> X-IsSubscribed: yes Eli Zaretskii wrote: > Gary Benson wrote: > > +library. This exercises all libthread_db functionality used by GDB on > > "libthread_db" should be in @code or in @file. And please use > @value{GDBN} instead of a literal "GDB". Done. FWIW I used @code for "libthread_db", it's what's used in @ref{set libthread-db-search-path}. > > +GNU/Linux systems, and by extension also exercises the proc_service > > proc_service should be in @code. Done. Pedro Alves wrote: > This looks good to me too, except one thing. I think you end up > with duplicated test messages: [snip] > I suggest wrapping the several phases of the testcase in > with_test_prefix or proc_with_prefix. Something like: [snip] Done. I'd changed the obvious messages for v2 of the patch, Simon mentioned something about with_test_prefix but at the time it didn't seem worth wrapping things just for one or two messages so I did it manually. I hadn't realized there were messages emitted by the setup commands that were also coming out duplicated (because the setup commands were the same). So I've used with_test_prefix this time, and used the recipe you linked to check the output. > OK with me with that change. Cool, I pushed the updated patch (included below). Have a good weekend everyone! Cheers, Gary --- This commit adds two new commands which may be used to test thread debugging libraries used by GDB: * "maint check libthread-db" tests the thread debugging library GDB is using for the current inferior. * "maint set/show check-libthread-db" selects whether libthread_db tests should be run automatically as libthread_db is auto-loaded. The default is to not run tests automatically. The test itself is a basic integrity check exercising all libthread_db functions used by GDB on GNU/Linux systems. By extension this also exercises the proc_service functions provided by GDB that libthread_db uses. This functionality is useful for NPTL developers and libthread_db developers. It could also prove useful investigating bugs reported against GDB where the thread debugging library or GDB's proc_service layer is suspect. gdb/ChangeLog: * linux-thread-db.c (valprint.h): New include. (struct check_thread_db_info): New structure. (check_thread_db_on_load, tdb_testinfo): New static globals. (check_thread_db, check_thread_db_callback): New functions. (try_thread_db_load_1): Run integrity checks if requested. (maintenance_check_libthread_db): New function. (_initialize_thread_db): Register "maint check libthread-db" and "maint set/show check-libthread-db". * NEWS: Mention the above new commands. gdb/doc/ChangeLog: * gdb.texinfo (Maintenance Commands): Document "maint check libthread-db" and "maint set/show check-libthread-db". gdb/testsuite/ChangeLog: * gdb.threads/check-libthread-db.exp: New file. * gdb.threads/check-libthread-db.c: Likewise. --- gdb/ChangeLog | 12 + gdb/NEWS | 10 + gdb/doc/ChangeLog | 5 + gdb/doc/gdb.texinfo | 20 ++ gdb/linux-thread-db.c | 291 +++++++++++++++++++++++ gdb/testsuite/ChangeLog | 5 + gdb/testsuite/gdb.threads/check-libthread-db.c | 67 ++++++ gdb/testsuite/gdb.threads/check-libthread-db.exp | 114 +++++++++ 8 files changed, 524 insertions(+) create mode 100644 gdb/testsuite/gdb.threads/check-libthread-db.c create mode 100644 gdb/testsuite/gdb.threads/check-libthread-db.exp diff --git a/gdb/NEWS b/gdb/NEWS index 8fb6a2a..13da2f1 100644 --- a/gdb/NEWS +++ b/gdb/NEWS @@ -36,6 +36,16 @@ set|show record btrace cpu Controls the processor to be used for enabling errata workarounds for branch trace decode. +maint check libthread-db + Run integrity checks on the current inferior's thread debugging + library + +maint set check-libthread-db (on|off) +maint show check-libthread-db + Control whether to run integrity checks on inferior specific thread + debugging libraries as they are loaded. The default is not to + perform such checks. + * Python API ** Type alignment is now exposed via the "align" attribute of a gdb.Type. diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo index 4968b37..2c0ac33 100644 --- a/gdb/doc/gdb.texinfo +++ b/gdb/doc/gdb.texinfo @@ -35547,6 +35547,15 @@ modify XML target descriptions. Check that the target descriptions dynamically created by @value{GDBN} equal the descriptions created from XML files found in @var{dir}. +@kindex maint check libthread-db +@item maint check libthread-db +Run integrity checks on the current inferior's thread debugging +library. This exercises all @code{libthread_db} functionality used by +@value{GDBN} on GNU/Linux systems, and by extension also exercises the +@code{proc_service} functions provided by @value{GDBN} that +@code{libthread_db} uses. Note that parts of the test may be skipped +on some platforms when debugging core files. + @kindex maint print dummy-frames @item maint print dummy-frames Prints the contents of @value{GDBN}'s internal dummy-frame stack. @@ -35854,6 +35863,17 @@ number of blocks in the blockvector @end enumerate @end table +@kindex maint set check-libthread-db +@kindex maint show check-libthread-db +@item maint set check-libthread-db [on|off] +@itemx maint show check-libthread-db +Control whether @value{GDBN} should run integrity checks on inferior +specific thread debugging libraries as they are loaded. The default +is not to perform such checks. If any check fails @value{GDBN} will +unload the library and continue searching for a suitable candidate as +described in @ref{set libthread-db-search-path}. For more information +about the tests, see @ref{maint check libthread-db}. + @kindex maint space @cindex memory used by commands @item maint space @var{value} diff --git a/gdb/linux-thread-db.c b/gdb/linux-thread-db.c index ccfd9e4..192e087 100644 --- a/gdb/linux-thread-db.c +++ b/gdb/linux-thread-db.c @@ -47,6 +47,7 @@ #include "nat/linux-namespaces.h" #include #include "common/pathstuff.h" +#include "valprint.h" /* GNU/Linux libthread_db support. @@ -117,6 +118,10 @@ static char *libthread_db_search_path; by the "set auto-load libthread-db" command. */ static int auto_load_thread_db = 1; +/* Set to non-zero if load-time libthread_db tests have been enabled + by the "maintenence set check-libthread-db" command. */ +static int check_thread_db_on_load = 0; + /* "show" command for the auto_load_thread_db configuration variable. */ static void @@ -534,6 +539,250 @@ dladdr_to_soname (const void *addr) return NULL; } +/* State for check_thread_db_callback. */ + +struct check_thread_db_info +{ + /* The libthread_db under test. */ + struct thread_db_info *info; + + /* True if progress should be logged. */ + bool log_progress; + + /* True if the callback was called. */ + bool threads_seen; + + /* Name of last libthread_db function called. */ + const char *last_call; + + /* Value returned by last libthread_db call. */ + td_err_e last_result; +}; + +static struct check_thread_db_info *tdb_testinfo; + +/* Callback for check_thread_db. */ + +static int +check_thread_db_callback (const td_thrhandle_t *th, void *arg) +{ + gdb_assert (tdb_testinfo != NULL); + tdb_testinfo->threads_seen = true; + +#define LOG(fmt, args...) \ + do \ + { \ + if (tdb_testinfo->log_progress) \ + { \ + debug_printf (fmt, ## args); \ + gdb_flush (gdb_stdlog); \ + } \ + } \ + while (0) + +#define CHECK_1(expr, args...) \ + do \ + { \ + if (!(expr)) \ + { \ + LOG (" ... FAIL!\n"); \ + error (args); \ + } \ + } \ + while (0) + +#define CHECK(expr) \ + CHECK_1 (expr, "(%s) == false", #expr) + +#define CALL_UNCHECKED(func, args...) \ + do \ + { \ + tdb_testinfo->last_call = #func; \ + tdb_testinfo->last_result \ + = tdb_testinfo->info->func ## _p (args); \ + } \ + while (0) + +#define CHECK_CALL() \ + CHECK_1 (tdb_testinfo->last_result == TD_OK, \ + _("%s failed: %s"), \ + tdb_testinfo->last_call, \ + thread_db_err_str (tdb_testinfo->last_result)) \ + +#define CALL(func, args...) \ + do \ + { \ + CALL_UNCHECKED (func, args); \ + CHECK_CALL (); \ + } \ + while (0) + + LOG (" Got thread"); + + /* Check td_ta_thr_iter passed consistent arguments. */ + CHECK (th != NULL); + CHECK (arg == (void *) tdb_testinfo); + CHECK (th->th_ta_p == tdb_testinfo->info->thread_agent); + + LOG (" %s", core_addr_to_string_nz ((CORE_ADDR) th->th_unique)); + + /* Check td_thr_get_info. */ + td_thrinfo_t ti; + CALL (td_thr_get_info, th, &ti); + + LOG (" => %d", ti.ti_lid); + + CHECK (ti.ti_ta_p == th->th_ta_p); + CHECK (ti.ti_tid == (thread_t) th->th_unique); + + /* Check td_ta_map_lwp2thr. */ + td_thrhandle_t th2; + memset (&th2, 23, sizeof (td_thrhandle_t)); + CALL_UNCHECKED (td_ta_map_lwp2thr, th->th_ta_p, ti.ti_lid, &th2); + + if (tdb_testinfo->last_result == TD_ERR && !target_has_execution) + { + /* Some platforms require execution for td_ta_map_lwp2thr. */ + LOG (_("; can't map_lwp2thr")); + } + else + { + CHECK_CALL (); + + LOG (" => %s", core_addr_to_string_nz ((CORE_ADDR) th2.th_unique)); + + CHECK (memcmp (th, &th2, sizeof (td_thrhandle_t)) == 0); + } + + /* Attempt TLS access. Assuming errno is TLS, this calls + thread_db_get_thread_local_address, which in turn calls + td_thr_tls_get_addr for live inferiors or td_thr_tlsbase + for core files. This test is skipped if the thread has + not been recorded; proceeding in that case would result + in the test having the side-effect of noticing threads + which seems wrong. + + Note that in glibc's libthread_db td_thr_tls_get_addr is + a thin wrapper around td_thr_tlsbase; this check always + hits the bulk of the code. + + Note also that we don't actually check any libthread_db + calls are made, we just assume they were; future changes + to how GDB accesses TLS could result in this passing + without exercising the calls it's supposed to. */ + ptid_t ptid = ptid_build (tdb_testinfo->info->pid, ti.ti_lid, 0); + struct thread_info *thread_info = find_thread_ptid (ptid); + if (thread_info != NULL && thread_info->priv != NULL) + { + LOG ("; errno"); + + scoped_restore_current_thread restore_current_thread; + switch_to_thread (ptid); + + expression_up expr = parse_expression ("(int) errno"); + struct value *val = evaluate_expression (expr.get ()); + + if (tdb_testinfo->log_progress) + { + struct value_print_options opts; + + get_user_print_options (&opts); + LOG (" = "); + value_print (val, gdb_stdlog, &opts); + } + } + + LOG (" ... OK\n"); + +#undef LOG +#undef CHECK_1 +#undef CHECK +#undef CALL_UNCHECKED +#undef CHECK_CALL +#undef CALL + + return 0; +} + +/* Run integrity checks on the dlopen()ed libthread_db described by + INFO. Returns true on success, displays a warning and returns + false on failure. Logs progress messages to gdb_stdlog during + the test if LOG_PROGRESS is true. */ + +static bool +check_thread_db (struct thread_db_info *info, bool log_progress) +{ + bool test_passed = true; + + if (log_progress) + debug_printf (_("Running libthread_db integrity checks:\n")); + + /* GDB avoids using td_ta_thr_iter wherever possible (see comment + in try_thread_db_load_1 below) so in order to test it we may + have to locate it ourselves. */ + td_ta_thr_iter_ftype *td_ta_thr_iter_p = info->td_ta_thr_iter_p; + if (td_ta_thr_iter_p == NULL) + { + void *thr_iter = verbose_dlsym (info->handle, "td_ta_thr_iter"); + if (thr_iter == NULL) + return 0; + + td_ta_thr_iter_p = (td_ta_thr_iter_ftype *) thr_iter; + } + + /* Set up the test state we share with the callback. */ + gdb_assert (tdb_testinfo == NULL); + struct check_thread_db_info tdb_testinfo_buf; + tdb_testinfo = &tdb_testinfo_buf; + + memset (tdb_testinfo, 0, sizeof (struct check_thread_db_info)); + tdb_testinfo->info = info; + tdb_testinfo->log_progress = log_progress; + + /* td_ta_thr_iter shouldn't be used on running processes. Note that + it's possible the inferior will stop midway through modifying one + of its thread lists, in which case the check will spuriously + fail. */ + linux_stop_and_wait_all_lwps (); + + TRY + { + td_err_e err = td_ta_thr_iter_p (info->thread_agent, + check_thread_db_callback, + tdb_testinfo, + TD_THR_ANY_STATE, + TD_THR_LOWEST_PRIORITY, + TD_SIGNO_MASK, + TD_THR_ANY_USER_FLAGS); + + if (err != TD_OK) + error (_("td_ta_thr_iter failed: %s"), thread_db_err_str (err)); + + if (!tdb_testinfo->threads_seen) + error (_("no threads seen")); + } + CATCH (except, RETURN_MASK_ERROR) + { + if (warning_pre_print) + fputs_unfiltered (warning_pre_print, gdb_stderr); + + exception_fprintf (gdb_stderr, except, + _("libthread_db integrity checks failed: ")); + + test_passed = false; + } + END_CATCH + + if (test_passed && log_progress) + debug_printf (_("libthread_db integrity checks passed.\n")); + + tdb_testinfo = NULL; + + linux_unstop_all_lwps (); + + return test_passed; +} + /* Attempt to initialize dlopen()ed libthread_db, described by INFO. Return 1 on success. Failure could happen if libthread_db does not have symbols we expect, @@ -627,6 +876,13 @@ try_thread_db_load_1 (struct thread_db_info *info) #undef TDB_DLSYM #undef CHK + /* Run integrity checks if requested. */ + if (check_thread_db_on_load) + { + if (!check_thread_db (info, libthread_db_debug)) + return 0; + } + if (info->td_ta_thr_iter_p == NULL) { struct lwp_info *lp; @@ -1658,6 +1914,24 @@ info_auto_load_libthread_db (const char *args, int from_tty) uiout->message (_("No auto-loaded libthread-db.\n")); } +/* Implement 'maintenance check libthread-db'. */ + +static void +maintenance_check_libthread_db (const char *args, int from_tty) +{ + int inferior_pid = ptid_get_pid (inferior_ptid); + struct thread_db_info *info; + + if (inferior_pid == 0) + error (_("No inferior running")); + + info = get_thread_db_info (inferior_pid); + if (info == NULL) + error (_("No libthread_db loaded")); + + check_thread_db (info, true); +} + void _initialize_thread_db (void) { @@ -1708,6 +1982,23 @@ This options has security implications for untrusted inferiors."), Usage: info auto-load libthread-db"), auto_load_info_cmdlist_get ()); + add_cmd ("libthread-db", class_maintenance, + maintenance_check_libthread_db, _("\ +Run integrity checks on the current inferior's libthread_db."), + &maintenancechecklist); + + add_setshow_boolean_cmd ("check-libthread-db", + class_maintenance, + &check_thread_db_on_load, _("\ +Set whether to check libthread_db at load time."), _("\ +Show whether to check libthread_db at load time."), _("\ +If enabled GDB will run integrity checks on inferior specific libthread_db\n\ +as they are loaded."), + NULL, + NULL, + &maintenance_set_cmdlist, + &maintenance_show_cmdlist); + /* Add ourselves to objfile event chain. */ gdb::observers::new_objfile.attach (thread_db_new_objfile); diff --git a/gdb/testsuite/gdb.threads/check-libthread-db.c b/gdb/testsuite/gdb.threads/check-libthread-db.c new file mode 100644 index 0000000..85a97a9 --- /dev/null +++ b/gdb/testsuite/gdb.threads/check-libthread-db.c @@ -0,0 +1,67 @@ +/* This testcase is part of GDB, the GNU debugger. + + Copyright 2017-2018 Free Software Foundation, Inc. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . */ + +#include +#include +#include +#include +#include +#include +#include + +static void +break_here (void) +{ +} + +static void * +thread_routine (void *arg) +{ + errno = 42; + + break_here (); + + while (1) + sleep (1); + + return NULL; +} + +int +main (int argc, char *argv) +{ + pthread_t the_thread; + int err; + + err = pthread_create (&the_thread, NULL, thread_routine, NULL); + if (err != 0) + { + fprintf (stderr, "pthread_create: %s (%d)\n", strerror (err), err); + exit (EXIT_FAILURE); + } + + errno = 23; + + err = pthread_join (the_thread, NULL); + if (err != 0) + { + fprintf (stderr, "pthread_join: %s (%d)\n", strerror (err), err); + exit (EXIT_FAILURE); + } + + exit (EXIT_SUCCESS); +} diff --git a/gdb/testsuite/gdb.threads/check-libthread-db.exp b/gdb/testsuite/gdb.threads/check-libthread-db.exp new file mode 100644 index 0000000..ddd9c27 --- /dev/null +++ b/gdb/testsuite/gdb.threads/check-libthread-db.exp @@ -0,0 +1,114 @@ +# Copyright 2017-2018 Free Software Foundation, Inc. + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# This test only works for native processes on GNU/Linux. +if {[target_info gdb_protocol] != "" || ![istarget *-linux*]} { + continue +} + +standard_testfile + +if {[gdb_compile_pthreads "${srcdir}/${subdir}/${srcfile}" "${binfile}" \ + executable debug] != "" } { + return -1 +} + +with_test_prefix "user-initiated check" { + + # User-initiated check with libthread_db not loaded. + clean_restart ${binfile} + + gdb_test "maint show check-libthread-db" \ + "Whether to check libthread_db at load time is off." + + gdb_test_no_output "set stop-on-solib-events 1" + gdb_run_cmd + gdb_test "" \ + ".*Stopped due to shared library event.*no libraries added or removed.*" + + gdb_test "maint check libthread-db" \ + "No libthread_db loaded" \ + "no libpthread.so loaded" + + + # User-initiated check with NPTL uninitialized. + # libthread_db should fake a single thread with th_unique == NULL. + gdb_test "continue" \ + ".*Stopped due to shared library event.*Inferior loaded .*libpthread.*" + + gdb_test_sequence "maint check libthread-db" \ + "libpthread.so not initialized" { + "\[\r\n\]+Running libthread_db integrity checks:" + "\[\r\n\]+\[ \]+Got thread 0x0 => \[0-9\]+ => 0x0 ... OK" + "\[\r\n\]+libthread_db integrity checks passed." + } + + # User-initiated check with NPTL fully operational. + gdb_test_no_output "set stop-on-solib-events 0" + gdb_breakpoint break_here + gdb_continue_to_breakpoint break_here + + gdb_test_sequence "maint check libthread-db" \ + "libpthread.so fully initialized" { + "\[\r\n\]+Running libthread_db integrity checks:" + "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+; errno = 23 ... OK" + "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+; errno = 42 ... OK" + "\[\r\n\]+libthread_db integrity checks passed." + } +} + +with_test_prefix "automated load-time check" { + + # Automated load-time check with NPTL uninitialized. + with_test_prefix "libpthread.so not initialized" { + clean_restart ${binfile} + + gdb_test_no_output "maint set check-libthread-db 1" + gdb_test_no_output "set debug libthread-db 1" + gdb_breakpoint break_here + gdb_run_cmd + + gdb_test_sequence "" \ + "check debug libthread-db output" { + "\[\r\n\]+Running libthread_db integrity checks:" + "\[\r\n\]+\[ \]+Got thread 0x0 => \[0-9\]+ => 0x0 ... OK" + "\[\r\n\]+libthread_db integrity checks passed." + "\[\r\n\]+[Thread debugging using libthread_db enabled]" + } + } + + # Automated load-time check with NPTL fully operational. + with_test_prefix "libpthread.so fully initialized" { + clean_restart ${binfile} + + gdb_test_no_output "maint set check-libthread-db 1" + gdb_test_no_output "set debug libthread-db 1" + + set test_spawn_id [spawn_wait_for_attach $binfile] + set testpid [spawn_id_get_pid $test_spawn_id] + + gdb_test_sequence "attach $testpid" \ + "check debug libthread-db output" { + "\[\r\n\]+Running libthread_db integrity checks:" + "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+ ... OK" + "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+ ... OK" + "\[\r\n\]+libthread_db integrity checks passed." + "\[\r\n\]+[Thread debugging using libthread_db enabled]" + } + + gdb_exit + kill_wait_spawned_process $test_spawn_id + } +}