diff mbox series

[1/2] Make linux checkpoints work with multiple inferiors

Message ID 20240323203056.1793487-2-kevinb@redhat.com
State New
Headers
Series Make linux checkpoints work with multiple inferiors |

Checks

Context Check Description
linaro-tcwg-bot/tcwg_gdb_build--master-aarch64 success Testing passed
linaro-tcwg-bot/tcwg_gdb_build--master-arm success Testing passed
linaro-tcwg-bot/tcwg_gdb_check--master-arm success Testing passed
linaro-tcwg-bot/tcwg_gdb_check--master-aarch64 success Testing passed

Commit Message

Kevin Buettner March 23, 2024, 8:27 p.m. UTC 
  The current linux checkpoint code, most of which may be found in
linux-fork.c, is quite broken when attempting to use more than
one inferior.  Running GDB will show internal errors when starting
two inferiors, placing a checkpoint in one, then switching to
the other and doing one of the following commands, "restart",
"detach", "kill", or continue (to program exit).  Test cases
for two of those scenarios may be found in the bug:

    https://sourceware.org/bugzilla/show_bug.cgi?id=31065

I've tested for each of the scenarios and many more in the new
test case: gdb.multi/checkpoint-multi.exp.

I started off with the goal of fixing just those problems, and was
mostly successful with a much smaller patch, but doing "info
checkpoints" with more than one inferior didn't work correctly due to
some of the inferiors being in the wrong program space.  So I decided
to do the right thing and make the linux-fork code inferior aware.

Prior to this commit, the list of forks was being maintained in a
global named named 'fork_list'.  I turned this into a per-inferior
data structure.  There is now a registry key named
'checkpoint_inferior_data_key' along with a new function named
'get_checkpoint_inferior_data', but it is only called by a new
function named 'get_fork_list'.  In all functions where the global
'fork_list' was previously accessed directly, a local named
'fork_list' is used instead.  In each of those functions, it is
declared and initialized like this:

    auto &fork_list = get_fork_list (inf);

In some cases, except for the addition of an additional
'inferior *inf' parameter, the rest of the function stays the same.

The constructor for 'struct fork_info' has gained an additional
parameter.  In addition to passing the pid of the new fork, we now
also pass the fork identifier, fork_num, to the constructor.  This
integer is shown to the user in the "info checkpoints" command and
is provided by the user in commands which manipulate checkpoints,
i.e. 'restart' and 'delete checkpoint'.

Adding the 'fork_num' parameter to the constructor simplified some of
the initialization when making a new checkpoint.  In particular, code
in add_fork() which initialized the static global highest_fork_num to
0 when there was only one fork has been removed.  Instead,
highest_fork_num now starts at -1.  When creating a checkpoint for an
inferior for which no checkpoints currently exist,
checkpoint_command() will now allocate what used to always be the
zeroeth fork number early on.  (This was done to preserve the existing
behavior of having lower numbered entries appear in the checkpoint
list first.  A fork number '0' will even still appear when the first
checkpoint is created.) Later on in checkpoint_command, the
start-of-list entry is added using the checkpoint number allocated
earlier in the function.

The old code was set up so that fork numbers would be reused if all
checkpoints were deleted and then new checkpoints were created.  That
no longer happens - checkpoint numbers are no longer reused.  There is
more than one precedent for not reusing identifiers in GDB, e.g.
breakpoint numbers aren't reused when one is deleted; the same is true
for inferior numbers.  Does this require a documentation change?  I
think not since the manual makes no mention of this behavior.

The functions find_fork_ptid, find_fork_id, and find_fork_pid
used to return a pointer to a fork_info struct.  Each of these
now return a pair consisting of the pointer to a fork_info struct
in addition to a pointer to the inferior containing that checkpoint.
Each of these functions now has at least one call where it is
necessary to have access to both the fork_info struct as well as
the inferior to which it applies.

info_checkpoints_command used to simply iterate over the list of
forks (checkpoints), printing each one out.  It now needs to iterate
over all inferiors and, for those which have checkpoints, it needs
to iterate over the list of checkpoints in that inferior.  This
command prints out a '*' preceding the currently active checkpoint.
That's still the case, but it'll now also print out a '+' preceding
the current checkpoint in non-current inferiors.  E.g...

    (gdb) info checkpoints
    + 4 process 8996 (main process) at 0x401199, file hello.c, line 51
      5 process 9004 at 0x401199, file hello.c, line 51
      1 process 9001 (main process) at 0x401258, file goodbye.c, line 62
    + 6 process 9005 at 0x401258, file goodbye.c, line 62
    * 7 process 9006 (main process) at 0x40115c, file hangout.c, line 31
      9 process 9008 at 0x40115c, file hangout.c, line 31

This shows checkpoints in three inferiors.  Fork #7 is the active one
in the current inferior, while #4, and #6 are the active forks in the
other two inferiors.  I haven't updated the documentation since it
didn't mention this behavior in the first place.

linux_fork_context, called by restart_command, now contains code to
switch inferiors when the fork being restarted is in a different
inferior from the current one.  The scoped_switch_fork_info class now
also contains code for switching inferiors in both the constructor and
destructor.

gdb/linux-nat.c has a few changes.  All but one of them are related
to passing the inferior to one of the linux-fork functions.  But
one of the tests in linux_nat_target::detach has also changed in
a non-obvious way.  In attempting to determine whether to call
linux_fork_detach(), that code used to do:

  if (pid == inferior_ptid.pid () && forks_exist_p ())

It's been simplified to:

  if (forks_exist_p (inf))

I had added the 'pid == inferior_ptid.pid ()' condition in late 2023
while working on a detach bug.  It was kind of a hack to prevent
calling linux_fork_detach() when in a different inferior.  That's no
longer needed since the call to forks_exist_p does this directly -
i.e. it is now inferior-aware.

Finally, the header file 'linux-fork.h' has been updated to reflect
the fact that add_fork, linux_fork_killall, linux_fork_mourn_inferior,
linux_fork_detach, and forks_exist_p all now require that a pointer to
an inferior be passed to these functions.  Additionally (as mentioned
earlier), find_fork_pid now returns std::pair<fork_info *, inferior *>
instead 'of fork_info *'.

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=31065
---
 gdb/linux-fork.c                             | 361 ++++++----
 gdb/linux-fork.h                             |  13 +-
 gdb/linux-nat.c                              |  20 +-
 gdb/testsuite/gdb.multi/checkpoint-multi.exp | 668 +++++++++++++++++++
 4 files changed, 922 insertions(+), 140 deletions(-)
 create mode 100644 gdb/testsuite/gdb.multi/checkpoint-multi.exp

Comments

Pedro Alves March 26, 2024, 12:55 p.m. UTC | #1 
Hi Kevin,

On 2024-03-23 20:27, Kevin Buettner wrote:

> info_checkpoints_command used to simply iterate over the list of
> forks (checkpoints), printing each one out.  It now needs to iterate
> over all inferiors and, for those which have checkpoints, it needs
> to iterate over the list of checkpoints in that inferior.  This
> command prints out a '*' preceding the currently active checkpoint.
> That's still the case, but it'll now also print out a '+' preceding
> the current checkpoint in non-current inferiors.  E.g...
> 
>     (gdb) info checkpoints
>     + 4 process 8996 (main process) at 0x401199, file hello.c, line 51
>       5 process 9004 at 0x401199, file hello.c, line 51
>       1 process 9001 (main process) at 0x401258, file goodbye.c, line 62
>     + 6 process 9005 at 0x401258, file goodbye.c, line 62
>     * 7 process 9006 (main process) at 0x40115c, file hangout.c, line 31
>       9 process 9008 at 0x40115c, file hangout.c, line 31

This is very confusing output/design IMO.  Checkpoint numbers aren't sorted, nor is
there any indication of which inferior each checkpoint is for.

IMO, a much better interface, is to instead make "info checkpoints" multi-inferior-aware
like "info threads" is:

By making each inferior have its own checkpoint number space.  I.e., like threads have
a INF.THR id, checkpoints would have an INF.CHKP id.

And like you can suppress the inferior number when refering thread ids, i.e,.
"thread 1.1" and "thread 1" is the same thread if we're focused on inferior 1,
so would "checkpoint 1.1" and "checkpoint 1".

And with that, checkpoints for an inferior could always start at 0 (or 1) like today,
and also like we always restart the inferior thread ids at 1 when we restart an inferior.

So the example above would turn into:

     (gdb) info checkpoints
     + 1.0 process 8996 (main process) at 0x401199, file hello.c, line 51
       1.1 process 9004 at 0x401199, file hello.c, line 51
       2.0 process 9001 (main process) at 0x401258, file goodbye.c, line 62
     + 2.1 process 9005 at 0x401258, file goodbye.c, line 62
     * 3.0 process 9006 (main process) at 0x40115c, file hangout.c, line 31
       3.1 process 9008 at 0x40115c, file hangout.c, line 31

"restart 1" or "restart 3.1" would switch to checkpoint 3.1.
"restart 2.1" would switch to checkpoint 1 of inferior 2.

With this, if you create a new checkpoint for inferior 2, it gets numbered as
checkpoint 2.2 , irrespective of whether which inferior you last created a checkpoint for.

I'm also not sure whether "+" is the best interface to indicate which checkpoint is active.
I'd think we could do that with a separate column, like, e.g., a state column that indicates
which checkpoint is active, and maybe also which one is the main process:

     (gdb) info checkpoints
       Id  State  Target Id   Frame
       1.0  AM process 8996 at 0x401199, file hello.c, line 51
       1.1     process 9004 at 0x401199, file hello.c, line 51
       2.0  M  process 9001 at 0x401258, file goodbye.c, line 62
       2.1  A  process 9005 at 0x401258, file goodbye.c, line 62
     * 3.0  AM process 9006 at 0x40115c, file hangout.c, line 31
       3.1     process 9008 at 0x40115c, file hangout.c, line 31

Though really "main process" info is redundant, as it's always the one with checkpoint id 0:

     (gdb) info checkpoints
       Id  State  Target Id   Frame
       1.0  A  process 8996 at 0x401199, file hello.c, line 51
       1.1     process 9004 at 0x401199, file hello.c, line 51
       2.0     process 9001 at 0x401258, file goodbye.c, line 62
       2.1  A  process 9005 at 0x401258, file goodbye.c, line 62
     * 3.0  A  process 9006 at 0x40115c, file hangout.c, line 31
       3.1     process 9008 at 0x40115c, file hangout.c, line 31

Pedro Alves
diff mbox series

Patch

diff --git a/gdb/linux-fork.c b/gdb/linux-fork.c
index 0d92d61f0d2..8e066341b63 100644
--- a/gdb/linux-fork.c
+++ b/gdb/linux-fork.c
@@ -29,6 +29,7 @@ 
 #include "linux-nat.h"
 #include "gdbthread.h"
 #include "source.h"
+#include "progspace-and-thread.h"
 
 #include "nat/gdb_ptrace.h"
 #include "gdbsupport/gdb_wait.h"
@@ -38,11 +39,14 @@ 
 
 #include <list>
 
+/* Most recently assigned fork number.  Fork numbers are never reused.  */
+static int highest_fork_num = -1;
+
 /* Fork list data structure:  */
 struct fork_info
 {
-  explicit fork_info (pid_t pid)
-    : ptid (pid, pid)
+  explicit fork_info (pid_t pid, int fork_num)
+    : ptid (pid, pid), num (fork_num)
   {
   }
 
@@ -70,7 +74,7 @@  struct fork_info
   ptid_t parent_ptid = null_ptid;
 
   /* Convenient handle (GDB fork id).  */
-  int num = 0;
+  int num;
 
   /* Convenient for info fork, saves having to actually switch
      contexts.  */
@@ -84,22 +88,56 @@  struct fork_info
   int maxfd = 0;
 };
 
-static std::list<fork_info> fork_list;
-static int highest_fork_num;
+struct checkpoint_inferior_data
+{
+  std::list<fork_info> fork_list;
+};
+
+/* Per-inferior data key.  */
+
+static const registry<inferior>::key<checkpoint_inferior_data>
+  checkpoint_inferior_data_key;
+
+/* Fetch per-inferior checkpoint data.  It always returns a valid pointer
+   to a checkpoint_inferior_info struct.  */
+
+static struct checkpoint_inferior_data *
+get_checkpoint_inferior_data (struct inferior *inf)
+{
+  struct checkpoint_inferior_data *data;
+
+  data = checkpoint_inferior_data_key.get (inf);
+  if (data == nullptr)
+    data = checkpoint_inferior_data_key.emplace (inf);
+
+  return data;
+}
+
+/* Return a reference to the per-inferior fork list.  */
+
+static std::list<fork_info> &
+get_fork_list (inferior *inf)
+{
+  return get_checkpoint_inferior_data (inf)->fork_list;
+}
 
 /* Fork list methods:  */
 
 int
-forks_exist_p (void)
+forks_exist_p (inferior *inf)
 {
+  auto &fork_list = get_fork_list (inf);
+
   return !fork_list.empty ();
 }
 
 /* Return the last fork in the list.  */
 
 static struct fork_info *
-find_last_fork (void)
+find_last_fork (inferior *inf)
 {
+  auto &fork_list = get_fork_list (inf);
+
   if (fork_list.empty ())
     return NULL;
 
@@ -109,30 +147,29 @@  find_last_fork (void)
 /* Return true iff there's one fork in the list.  */
 
 static bool
-one_fork_p ()
+one_fork_p (inferior *inf)
 {
+  auto &fork_list = get_fork_list (inf);
+
   return fork_list.size () == 1;
 }
 
 /* Add a new fork to the internal fork list.  */
 
 void
-add_fork (pid_t pid)
+add_fork (pid_t pid, inferior *inf)
 {
-  fork_list.emplace_back (pid);
-
-  if (one_fork_p ())
-    highest_fork_num = 0;
+  auto &fork_list = get_fork_list (inf);
 
-  fork_info *fp = &fork_list.back ();
-  fp->num = ++highest_fork_num;
+  fork_list.emplace_back (pid, ++highest_fork_num);
 }
 
 static void
-delete_fork (ptid_t ptid)
+delete_fork (ptid_t ptid, inferior *inf)
 {
   linux_target->low_forget_process (ptid.pid ());
 
+  auto &fork_list = get_fork_list (inf);
   for (auto it = fork_list.begin (); it != fork_list.end (); ++it)
     if (it->ptid == ptid)
       {
@@ -142,55 +179,68 @@  delete_fork (ptid_t ptid)
 	   and if it is (hopefully!) the current inferior_ptid, then
 	   remove it, leaving the list empty -- we're now down to the
 	   default case of debugging a single process.  */
-	if (one_fork_p () && fork_list.front ().ptid == inferior_ptid)
+	if (one_fork_p (inf) && fork_list.front ().ptid == inferior_ptid)
 	  {
 	    /* Last fork -- delete from list and handle as solo
 	       process (should be a safe recursion).  */
-	    delete_fork (inferior_ptid);
+	    delete_fork (inferior_ptid, inf);
 	  }
 	return;
       }
 }
 
 /* Find a fork_info by matching PTID.  */
-static struct fork_info *
+static std::pair<fork_info *, inferior *>
 find_fork_ptid (ptid_t ptid)
 {
-  for (fork_info &fi : fork_list)
-    if (fi.ptid == ptid)
-      return &fi;
+  for (inferior *inf : all_inferiors ())
+    {
+      auto &fork_list = get_fork_list (inf);
+      for (fork_info &fi : fork_list)
+	if (fi.ptid == ptid)
+	  return { &fi, inf };
+    }
 
-  return NULL;
+  return { nullptr, nullptr };
 }
 
 /* Find a fork_info by matching ID.  */
-static struct fork_info *
+static std::pair<fork_info *, inferior *>
 find_fork_id (int num)
 {
-  for (fork_info &fi : fork_list)
-    if (fi.num == num)
-      return &fi;
+  for (inferior *inf : all_inferiors ())
+    {
+      auto &fork_list = get_fork_list (inf);
+      for (fork_info &fi : fork_list)
+	if (fi.num == num)
+	  return { &fi, inf };
+    }
 
-  return NULL;
+  return { nullptr, nullptr };
 }
 
 /* Find a fork_info by matching pid.  */
-extern struct fork_info *
+extern std::pair<fork_info *, inferior *>
 find_fork_pid (pid_t pid)
 {
-  for (fork_info &fi : fork_list)
-    if (pid == fi.ptid.pid ())
-      return &fi;
+  for (inferior *inf : all_inferiors ())
+    {
+      auto &fork_list = get_fork_list (inf);
+      for (fork_info &fi : fork_list)
+	if (pid == fi.ptid.pid ())
+	  return { &fi, inf };
+    }
 
-  return NULL;
+  return { nullptr, nullptr };
 }
 
+/* Find a fork's ptid given it's number NUM.  */
 static ptid_t
 fork_id_to_ptid (int num)
 {
-  struct fork_info *fork = find_fork_id (num);
-  if (fork)
-    return fork->ptid;
+  struct fork_info *fi = find_fork_id (num).first;
+  if (fi != nullptr)
+    return fi->ptid;
   else
     return ptid_t (-1);
 }
@@ -297,7 +347,7 @@  fork_save_infrun_state (struct fork_info *fp)
 /* Kill 'em all, let God sort 'em out...  */
 
 void
-linux_fork_killall (void)
+linux_fork_killall (inferior *inf)
 {
   /* Walk list and kill every pid.  No need to treat the
      current inferior_ptid as special (we do not return a
@@ -305,6 +355,7 @@  linux_fork_killall (void)
      or a parent, so may get a SIGCHLD from a previously
      killed child.  Wait them all out.  */
 
+  auto &fork_list = get_fork_list (inf);
   for (fork_info &fi : fork_list)
     {
       pid_t pid = fi.ptid.pid ();
@@ -330,7 +381,7 @@  linux_fork_killall (void)
    first available.  */
 
 void
-linux_fork_mourn_inferior (void)
+linux_fork_mourn_inferior (inferior *inf)
 {
   struct fork_info *last;
   int status;
@@ -344,21 +395,21 @@  linux_fork_mourn_inferior (void)
   /* OK, presumably inferior_ptid is the one who has exited.
      We need to delete that one from the fork_list, and switch
      to the next available fork.  */
-  delete_fork (inferior_ptid);
+  delete_fork (inferior_ptid, inf);
 
   /* There should still be a fork - if there's only one left,
      delete_fork won't remove it, because we haven't updated
      inferior_ptid yet.  */
-  gdb_assert (!fork_list.empty ());
+  gdb_assert (!get_fork_list (inf).empty ());
 
-  last = find_last_fork ();
+  last = find_last_fork (inf);
   fork_load_infrun_state (last);
   gdb_printf (_("[Switching to %s]\n"),
 	      target_pid_to_str (inferior_ptid).c_str ());
 
   /* If there's only one fork, switch back to non-fork mode.  */
-  if (one_fork_p ())
-    delete_fork (inferior_ptid);
+  if (one_fork_p (inf))
+    delete_fork (inferior_ptid, inf);
 }
 
 /* The current inferior_ptid is being detached, but there are other
@@ -366,7 +417,7 @@  linux_fork_mourn_inferior (void)
    the first available.  */
 
 void
-linux_fork_detach (int from_tty, lwp_info *lp)
+linux_fork_detach (int from_tty, lwp_info *lp, inferior *inf)
 {
   gdb_assert (lp != nullptr);
   gdb_assert (lp->ptid == inferior_ptid);
@@ -385,11 +436,12 @@  linux_fork_detach (int from_tty, lwp_info *lp)
 	       target_pid_to_str (inferior_ptid).c_str ());
     }
 
-  delete_fork (inferior_ptid);
+  delete_fork (inferior_ptid, inf);
 
   /* There should still be a fork - if there's only one left,
      delete_fork won't remove it, because we haven't updated
      inferior_ptid yet.  */
+  auto &fork_list = get_fork_list (inf);
   gdb_assert (!fork_list.empty ());
 
   fork_load_infrun_state (&fork_list.front ());
@@ -399,8 +451,8 @@  linux_fork_detach (int from_tty, lwp_info *lp)
 		target_pid_to_str (inferior_ptid).c_str ());
 
   /* If there's only one fork, switch back to non-fork mode.  */
-  if (one_fork_p ())
-    delete_fork (inferior_ptid);
+  if (one_fork_p (inf))
+    delete_fork (inferior_ptid, inf);
 }
 
 /* Temporarily switch to the infrun state stored on the fork_info
@@ -413,19 +465,26 @@  class scoped_switch_fork_info
   /* Switch to the infrun state held on the fork_info identified by
      PPTID.  If PPTID is the current inferior then no switch is done.  */
   explicit scoped_switch_fork_info (ptid_t pptid)
-    : m_oldfp (nullptr)
+    : m_oldfp (nullptr), m_oldinf (nullptr)
   {
     if (pptid != inferior_ptid)
       {
-	struct fork_info *newfp = nullptr;
-
 	/* Switch to pptid.  */
-	m_oldfp = find_fork_ptid (inferior_ptid);
+	auto [oldfp, oldinf] = find_fork_ptid (inferior_ptid);
+	m_oldfp = oldfp;
 	gdb_assert (m_oldfp != nullptr);
-	newfp = find_fork_ptid (pptid);
+	auto [newfp, newinf]  = find_fork_ptid (pptid);
 	gdb_assert (newfp != nullptr);
 	fork_save_infrun_state (m_oldfp);
 	remove_breakpoints ();
+
+	if (oldinf != newinf)
+	  {
+	    thread_info *tp = any_thread_of_inferior (newinf);
+	    switch_to_thread (tp);
+	    m_oldinf = oldinf;
+	  }
+
 	fork_load_infrun_state (newfp);
 	insert_breakpoints ();
       }
@@ -435,12 +494,17 @@  class scoped_switch_fork_info
      didn't need to switch states, then nothing is done here either.  */
   ~scoped_switch_fork_info ()
   {
-    if (m_oldfp != nullptr)
+    if (m_oldinf != nullptr || m_oldfp != nullptr)
       {
 	/* Switch back to inferior_ptid.  */
 	try
 	  {
 	    remove_breakpoints ();
+	    if (m_oldinf != nullptr)
+	      {
+		thread_info *tp = any_thread_of_inferior (m_oldinf);
+		switch_to_thread (tp);
+	      }
 	    fork_load_infrun_state (m_oldfp);
 	    insert_breakpoints ();
 	  }
@@ -472,6 +536,7 @@  class scoped_switch_fork_info
      we were already in the desired state, and nothing needs to be
      restored.  */
   struct fork_info *m_oldfp;
+  inferior *m_oldinf;
 };
 
 static int
@@ -514,7 +579,6 @@  static void
 delete_checkpoint_command (const char *args, int from_tty)
 {
   ptid_t ptid, pptid;
-  struct fork_info *fi;
 
   if (!args || !*args)
     error (_("Requires argument (checkpoint id to delete)"));
@@ -523,21 +587,30 @@  delete_checkpoint_command (const char *args, int from_tty)
   if (ptid == minus_one_ptid)
     error (_("No such checkpoint id, %s"), args);
 
-  if (ptid == inferior_ptid)
-    error (_("\
+  auto [fi, inf] = find_fork_ptid (ptid);
+  gdb_assert (fi != nullptr);
+  pptid = fi->parent_ptid;
+
+  thread_info *inf_thr = any_thread_of_inferior (inf);
+  gdb_assert (inf_thr != nullptr);
+  if (ptid == inf_thr->ptid)
+    {
+      if (inf == current_inferior ())
+	error (_("\
 Please switch to another checkpoint before deleting the current one"));
+      else
+	error (_("\
+Please switch to another checkpoint before deleting the current one\n\
+in inferior %d"), inf->num);
+    }
 
   if (ptrace (PTRACE_KILL, ptid.pid (), 0, 0))
     error (_("Unable to kill pid %s"), target_pid_to_str (ptid).c_str ());
 
-  fi = find_fork_ptid (ptid);
-  gdb_assert (fi);
-  pptid = fi->parent_ptid;
-
   if (from_tty)
     gdb_printf (_("Killed %s\n"), target_pid_to_str (ptid).c_str ());
 
-  delete_fork (ptid);
+  delete_fork (ptid, inf);
 
   if (pptid == null_ptid)
     {
@@ -555,7 +628,7 @@  Please switch to another checkpoint before deleting the current one"));
      If fi->parent_ptid is a part of lwp and it is stopped, waitpid the
      ptid.  */
   thread_info *parent = linux_target->find_thread (pptid);
-  if ((parent == NULL && find_fork_ptid (pptid))
+  if ((parent == NULL && find_fork_ptid (pptid).first != nullptr)
       || (parent != NULL && parent->state == THREAD_STOPPED))
     {
       if (inferior_call_waitpid (pptid, ptid.pid ()))
@@ -586,7 +659,7 @@  Please switch to another checkpoint before detaching the current one"));
   if (from_tty)
     gdb_printf (_("Detached %s\n"), target_pid_to_str (ptid).c_str ());
 
-  delete_fork (ptid);
+  delete_fork (ptid, current_inferior ());
 }
 
 /* Print information about currently known checkpoints.  */
@@ -595,57 +668,72 @@  static void
 info_checkpoints_command (const char *arg, int from_tty)
 {
   struct gdbarch *gdbarch = get_current_arch ();
+  struct inferior *cur_inf = current_inferior ();
   int requested = -1;
   bool printed = false;
 
   if (arg && *arg)
     requested = (int) parse_and_eval_long (arg);
 
-  for (const fork_info &fi : fork_list)
-    {
-      if (requested > 0 && fi.num != requested)
-	continue;
-      printed = true;
-
-      bool is_current = fi.ptid == inferior_ptid;
-      if (is_current)
-	gdb_printf ("* ");
-      else
-	gdb_printf ("  ");
-
-      gdb_printf ("%d %s", fi.num, target_pid_to_str (fi.ptid).c_str ());
-      if (fi.num == 0)
-	gdb_printf (_(" (main process)"));
 
-      if (is_current && inferior_thread ()->state == THREAD_RUNNING)
-	{
-	  gdb_printf (_(" <running>\n"));
-	  continue;
-	}
+  for (inferior *inf : all_inferiors ())
+    {
+      scoped_restore_current_pspace_and_thread restore_pspace_thread;
+      switch_to_program_space_and_thread (inf->pspace);
 
-      gdb_printf (_(" at "));
-      ULONGEST pc
-	= (is_current
-	   ? regcache_read_pc (get_thread_regcache (inferior_thread ()))
-	   : fi.pc);
-      gdb_puts (paddress (gdbarch, pc));
-
-      symtab_and_line sal = find_pc_line (pc, 0);
-      if (sal.symtab)
-	gdb_printf (_(", file %s"),
-		    symtab_to_filename_for_display (sal.symtab));
-      if (sal.line)
-	gdb_printf (_(", line %d"), sal.line);
-      if (!sal.symtab && !sal.line)
+      auto &fork_list = get_fork_list (inf);
+      for (const fork_info &fi : fork_list)
 	{
-	  struct bound_minimal_symbol msym;
-
-	  msym = lookup_minimal_symbol_by_pc (pc);
-	  if (msym.minsym)
-	    gdb_printf (", <%s>", msym.minsym->linkage_name ());
+	  if (requested > 0 && fi.num != requested)
+	    continue;
+	  printed = true;
+
+	  thread_info *t = any_thread_of_inferior (inf);
+	  bool is_current = fi.ptid == t->ptid;
+	  if (is_current)
+	    {
+	      if (cur_inf == inf)
+		gdb_printf ("* ");
+	      else
+		gdb_printf ("+ ");
+	    }
+	  else
+	    gdb_printf ("  ");
+
+	  gdb_printf ("%d %s", fi.num, target_pid_to_str (fi.ptid).c_str ());
+	  if (&fi == &fork_list.front ())
+	    gdb_printf (_(" (main process)"));
+
+	  if (is_current && t->state == THREAD_RUNNING)
+	    {
+	      gdb_printf (_(" <running>\n"));
+	      continue;
+	    }
+
+	  gdb_printf (_(" at "));
+	  ULONGEST pc
+	    = (is_current
+	       ? regcache_read_pc (get_thread_regcache (t))
+	       : fi.pc);
+	  gdb_puts (paddress (gdbarch, pc));
+
+	  symtab_and_line sal = find_pc_line (pc, 0);
+	  if (sal.symtab)
+	    gdb_printf (_(", file %s"),
+			symtab_to_filename_for_display (sal.symtab));
+	  if (sal.line)
+	    gdb_printf (_(", line %d"), sal.line);
+	  if (!sal.symtab && !sal.line)
+	    {
+	      struct bound_minimal_symbol msym;
+
+	      msym = lookup_minimal_symbol_by_pc (pc);
+	      if (msym.minsym)
+		gdb_printf (", <%s>", msym.minsym->linkage_name ());
+	    }
+
+	  gdb_putc ('\n');
 	}
-
-      gdb_putc ('\n');
     }
 
   if (!printed)
@@ -690,7 +778,6 @@  checkpoint_command (const char *args, int from_tty)
   struct target_waitstatus last_target_waitstatus;
   ptid_t last_target_ptid;
   struct value *fork_fn = NULL, *ret;
-  struct fork_info *fp;
   pid_t retpid;
 
   if (!target_has_execution ()) 
@@ -711,6 +798,10 @@  checkpoint_command (const char *args, int from_tty)
   if (!fork_fn)
     error (_("checkpoint: can't find fork function in inferior."));
 
+  int first_num = 0;
+  if (!forks_exist_p (current_inferior ()))
+    first_num = ++highest_fork_num;
+
   gdbarch = fork_objf->arch ();
   ret = value_from_longest (builtin_type (gdbarch)->builtin_int, 0);
 
@@ -728,7 +819,7 @@  checkpoint_command (const char *args, int from_tty)
   retpid = value_as_long (ret);
   get_last_target_status (nullptr, &last_target_ptid, &last_target_waitstatus);
 
-  fp = find_fork_pid (retpid);
+  auto [fp, inf] = find_fork_pid (retpid);
 
   if (from_tty)
     {
@@ -749,12 +840,14 @@  checkpoint_command (const char *args, int from_tty)
   if (!fp)
     error (_("Failed to find new fork"));
 
-  if (one_fork_p ())
+  if (one_fork_p (inf))
     {
       /* Special case -- if this is the first fork in the list (the
-	 list was hitherto empty), then add inferior_ptid first, as a
-	 special zeroeth fork id.  */
-      fork_list.emplace_front (inferior_ptid.pid ());
+	 list was hitherto empty), then add inferior_ptid first, using
+	 the fork number allocated earlier - see above assignment to
+	 FIRST_NUM.  */
+      auto &fork_list = get_fork_list (inf);
+      fork_list.emplace_front (inferior_ptid.pid (), first_num);
     }
 
   fork_save_infrun_state (fp);
@@ -762,40 +855,56 @@  checkpoint_command (const char *args, int from_tty)
 }
 
 static void
-linux_fork_context (struct fork_info *newfp, int from_tty)
+linux_fork_context (struct fork_info *newfp, int from_tty, inferior *newinf)
 {
-  /* Now we attempt to switch processes.  */
-  struct fork_info *oldfp;
+  bool inferior_changed = false;
 
+  /* Now we attempt to switch processes.  */
   gdb_assert (newfp != NULL);
 
-  oldfp = find_fork_ptid (inferior_ptid);
-  gdb_assert (oldfp != NULL);
+  if (newinf != current_inferior ())
+    {
+      thread_info *tp = any_thread_of_inferior (newinf);
+      switch_to_thread (tp);
+      inferior_changed = true;
+    }
 
-  fork_save_infrun_state (oldfp);
-  remove_breakpoints ();
-  fork_load_infrun_state (newfp);
-  insert_breakpoints ();
+  auto [oldfp, oldinf] = find_fork_ptid (inferior_ptid);
+  gdb_assert (oldfp != NULL);
 
-  gdb_printf (_("Switching to %s\n"),
-	      target_pid_to_str (inferior_ptid).c_str ());
+  if (oldfp == newfp)
+    {
+      if (!inferior_changed)
+	error (_("Already using checkpoint %d\n"), oldfp->num);
+    }
+  else
+    {
+      fork_save_infrun_state (oldfp);
+      remove_breakpoints ();
+      fork_load_infrun_state (newfp);
+      insert_breakpoints ();
+      if (!inferior_changed)
+	gdb_printf (_("Switching to %s\n"),
+		    target_pid_to_str (inferior_ptid).c_str ());
+    }
 
-  print_stack_frame (get_selected_frame (NULL), 1, SRC_AND_LOC, 1);
+  notify_user_selected_context_changed
+    (inferior_changed ? (USER_SELECTED_INFERIOR | USER_SELECTED_FRAME)
+		      : USER_SELECTED_FRAME);
 }
 
 /* Switch inferior process (checkpoint) context, by checkpoint id.  */
 static void
 restart_command (const char *args, int from_tty)
 {
-  struct fork_info *fp;
-
   if (!args || !*args)
     error (_("Requires argument (checkpoint id to restart)"));
 
-  if ((fp = find_fork_id (parse_and_eval_long (args))) == NULL)
+  auto [fp, inf] = find_fork_id (parse_and_eval_long (args));
+  if (fp == nullptr)
     error (_("Not found: checkpoint id %s"), args);
 
-  linux_fork_context (fp, from_tty);
+  linux_fork_context (fp, from_tty, inf);
 }
 
 void _initialize_linux_fork ();
diff --git a/gdb/linux-fork.h b/gdb/linux-fork.h
index c553aaf0740..e8ed37a4118 100644
--- a/gdb/linux-fork.h
+++ b/gdb/linux-fork.h
@@ -22,12 +22,13 @@ 
 
 struct fork_info;
 struct lwp_info;
-extern void add_fork (pid_t);
-extern struct fork_info *find_fork_pid (pid_t);
-extern void linux_fork_killall (void);
-extern void linux_fork_mourn_inferior (void);
-extern void linux_fork_detach (int, lwp_info *);
-extern int forks_exist_p (void);
+class inferior;
+extern void add_fork (pid_t, inferior *inf);
+extern std::pair<fork_info *, inferior *> find_fork_pid (pid_t);
+extern void linux_fork_killall (inferior *inf);
+extern void linux_fork_mourn_inferior (inferior *inf);
+extern void linux_fork_detach (int, lwp_info *, inferior *inf);
+extern int forks_exist_p (inferior *inf);
 extern int linux_fork_checkpointing_p (int);
 
 #endif /* LINUX_FORK_H */
diff --git a/gdb/linux-nat.c b/gdb/linux-nat.c
index 3ba072bc8d7..5181aa2421d 100644
--- a/gdb/linux-nat.c
+++ b/gdb/linux-nat.c
@@ -1553,13 +1553,13 @@  linux_nat_target::detach (inferior *inf, int from_tty)
   gdb_assert (num_lwps (pid) == 1
 	      || (target_is_non_stop_p () && num_lwps (pid) == 0));
 
-  if (pid == inferior_ptid.pid () && forks_exist_p ())
+  if (forks_exist_p (inf))
     {
       /* Multi-fork case.  The current inferior_ptid is being detached
 	 from, but there are other viable forks to debug.  Detach from
 	 the current fork, and context-switch to the first
 	 available.  */
-      linux_fork_detach (from_tty, find_lwp_pid (ptid_t (pid)));
+      linux_fork_detach (from_tty, find_lwp_pid (ptid_t (pid)), inf);
     }
   else
     {
@@ -2077,8 +2077,12 @@  linux_handle_extended_wait (struct lwp_info *lp, int status)
 	  detach_breakpoints (ptid_t (new_pid, new_pid));
 
 	  /* Retain child fork in ptrace (stopped) state.  */
-	  if (!find_fork_pid (new_pid))
-	    add_fork (new_pid);
+	  if (find_fork_pid (new_pid).first == nullptr)
+	    {
+	      struct inferior *inf = find_inferior_ptid (linux_target,
+	                                                 lp->ptid);
+	      add_fork (new_pid, inf);
+	    }
 
 	  /* Report as spurious, so that infrun doesn't want to follow
 	     this fork.  We're actually doing an infcall in
@@ -3758,8 +3762,8 @@  linux_nat_target::kill ()
      parent will be sleeping if this is a vfork.  */
   iterate_over_lwps (pid_ptid, kill_unfollowed_child_callback);
 
-  if (forks_exist_p ())
-    linux_fork_killall ();
+  if (forks_exist_p (current_inferior ()))
+    linux_fork_killall (current_inferior ());
   else
     {
       /* Stop all threads before killing them, since ptrace requires
@@ -3790,14 +3794,14 @@  linux_nat_target::mourn_inferior ()
 
   close_proc_mem_file (pid);
 
-  if (! forks_exist_p ())
+  if (! forks_exist_p (current_inferior ()))
     /* Normal case, no other forks available.  */
     inf_ptrace_target::mourn_inferior ();
   else
     /* Multi-fork case.  The current inferior_ptid has exited, but
        there are other viable forks to debug.  Delete the exiting
        one and context-switch to the first available.  */
-    linux_fork_mourn_inferior ();
+    linux_fork_mourn_inferior (current_inferior ());
 
   /* Let the arch-specific native code know this process is gone.  */
   linux_target->low_forget_process (pid);
diff --git a/gdb/testsuite/gdb.multi/checkpoint-multi.exp b/gdb/testsuite/gdb.multi/checkpoint-multi.exp
new file mode 100644
index 00000000000..98f6e0ae933
--- /dev/null
+++ b/gdb/testsuite/gdb.multi/checkpoint-multi.exp
@@ -0,0 +1,668 @@ 
+# Copyright 2009-2024 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This file tests various scenarios involving multiple inferiors
+# and the checkpoint command.
+
+# Checkpoint support works only on Linux.
+require {istarget "*-*-linux*"}
+
+# Checkpoint support is implemented for the (Linux) native target,
+# which is indicated when [target_info gdb_protocol] returns the empty
+# string.
+if { [target_info gdb_protocol] != "" } {
+    return
+}
+
+set testfile "checkpoint-multi"
+
+set exec1 "hello"
+set srcfile1 ${exec1}.c
+set binfile1 [standard_output_file ${exec1}]
+
+set exec2 "goodbye"
+set srcfile2 ${exec2}.c
+set binfile2 [standard_output_file ${exec2}]
+
+set exec3 "hangout"
+set srcfile3 ${exec3}.c
+set binfile3 [standard_output_file ${exec3}]
+
+if { [build_executable ${testfile}.exp ${exec1} "${srcfile1}" {debug}] == -1 } {
+    return -1
+}
+
+if { [build_executable ${testfile}.exp ${exec2} "${srcfile2}" {debug}] == -1 } {
+    return -1
+}
+
+if { [build_executable ${testfile}.exp ${exec3} "${srcfile3}" {debug}] == -1 } {
+    return -1
+}
+
+# Start two inferiors, place a catchpoint on inferior 2, but switch
+# back to inferior 1.
+proc start_2_inferiors_catchpoint_on_inf_2 {} {
+    clean_restart $::exec1
+
+    # Start inferior 1.
+    if {[gdb_start_cmd] < 0} {
+	fail "start first inferior"
+    } else {
+	gdb_test "" "main.*" "start first inferior"
+    }
+
+    # Add a new inferior and exec into it.
+    gdb_test "add-inferior -exec $::binfile2" \
+	"Added inferior 2.*" \
+	"add inferior 2 with -exec $::exec2"
+
+    # Check that we have multiple inferiors.
+    gdb_test "info inferiors" \
+	"Executable.*$::exec1.*$::exec2.*"
+
+    # Switch to inferior 2.
+    gdb_test "inferior 2" \
+	"Switching to inferior 2.*$::exec2.*"
+
+    # Start inferior 2:
+    if {[gdb_start_cmd] < 0} {
+	fail "start second inferior"
+    } else {
+	gdb_test "" "main.*" "start second inferior"
+    }
+
+    # Set a checkpoint in inferior 2
+    gdb_test "checkpoint" "checkpoint 1: fork returned pid $::decimal.*"
+
+    # Step one line in inferior 2.
+    gdb_test "step" "glob = 46;"
+
+    # Switch back to inferior 1.
+    gdb_test "inferior 1" "Switching to inferior 1.*$::exec1.*"
+}
+
+# Start two inferiors, place a catchpoint on inferior 2, but switch
+# back to inferior 1.  This is like the one above, except that it
+# swaps the executables loaded into inferior 1 and inferior 2.	This
+# is important for being able to test "continue to exit".  (Because... 
+# hello.c has an infinite loop, but goodbye.c doesn't.	In order to
+# test "continue to exit", we need to continue in an executable which
+# will actually exit.)
+
+proc start_2_inferiors_catchpoint_on_inf_2_alt {} {
+    clean_restart $::exec2
+
+    # Start inferior 1.
+    if {[gdb_start_cmd] < 0} {
+	fail "start first inferior"
+    } else {
+	gdb_test "" "main.*" "start first inferior"
+    }
+
+    # Add a new inferior and exec exec1 into it.
+    gdb_test "add-inferior -exec $::binfile1" \
+	"Added inferior 2.*" \
+	"add inferior 2 with -exec $::exec1"
+
+    # Check that we have two inferiors.
+    gdb_test "info inferiors" \
+	"Executable.*$::exec2.*$::exec1.*"
+
+    # Switch to inferior 2.
+    gdb_test "inferior 2" \
+	"Switching to inferior 2.*$::exec1.*"
+
+    # Start inferior 2:
+    if {[gdb_start_cmd] < 0} {
+	fail "start second inferior"
+    } else {
+	gdb_test "" "main.*" "start second inferior"
+    }
+
+    # Set a checkpoint in inferior 2
+    gdb_test "checkpoint" "checkpoint 1: fork returned pid $::decimal.*"
+
+    # next one line in inferior 2.
+    gdb_test "next" "bar\\(\\).*"
+
+    # Switch back to inferior 1.
+    gdb_test "inferior 1" "Switching to inferior 1.*$::exec2.*"
+}
+
+with_test_prefix "check detach on non-checkpointed inferior" {
+    start_2_inferiors_catchpoint_on_inf_2
+    gdb_test "detach" "Detaching from program.*$::exec1.*Inferior 1.*detached.*"
+}
+
+with_test_prefix "check kill on non-checkpointed inferior" {
+    start_2_inferiors_catchpoint_on_inf_2
+    gdb_test "kill" "" "kill non-checkpointed inferior" \
+	     "Kill the program being debugged.*y or n. $" "y"
+}
+
+with_test_prefix "check restart 0 on non-checkpointed inferior" {
+    start_2_inferiors_catchpoint_on_inf_2
+    gdb_test "restart 0" "Switching to inferior 2.*?goodbye.*?#0 +mailand .*?glob = 46;.*"
+}
+
+with_test_prefix "check restart 1 on non-checkpointed inferior" {
+    start_2_inferiors_catchpoint_on_inf_2
+    gdb_test "restart 1" "Switching to inferior 2.*?goodbye.*?#0 +main .*?mailand\\(\\);.*"
+}
+
+with_test_prefix "check continue to exit on non-checkpointed inferior" {
+    start_2_inferiors_catchpoint_on_inf_2_alt
+    gdb_test "continue" "Inferior 1.*? exited normally.*"
+}
+
+set thr_or_proc "(?:process $::decimal|Thread $::hex \\(LWP $::decimal\\))"
+set main_proc "\\(main process\\)"
+set hello_c "hello\\.c"
+set goodbye_c "goodbye\\.c"
+set hangout_c "hangout\\.c"
+
+with_test_prefix "two inferiors with checkpoints" {
+    start_2_inferiors_catchpoint_on_inf_2
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 1"
+    gdb_test "checkpoint" "checkpoint 3: fork returned pid $::decimal.*" \
+	     "checkpoint in inferior 1"
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\* 2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 2"
+    gdb_test "restart 0" \
+	     "\\\[Switching to inferior 2.*?mailand.*?glob = 46;.*"
+    gdb_test "next" "\}"
+
+    gdb_test "restart 1" "^Switching to $thr_or_proc.*?#0  main \\(\\) at.*?$goodbye_c.*mailand\\(\\);"
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 3"
+
+    # Doing "info_checkpoints" twice in a row might seem pointless,
+    # but during work on making the checkpoint code inferior aware,
+    # there was a point at which doing it twice in a row did not
+    # produce the same output.
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 4"
+
+    # Switch back to checkpoint 0; again, there should be no
+    # "Switching to inferior" message.
+    gdb_test "restart 0" "^Switching to $thr_or_proc.*?#0  mailand \\(\\) at.*?$goodbye_c.*\}" \
+	     "restart 0 #2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 5"
+
+    # Switch to checkpoint 3; this time, we should see a "Switching to
+    # inferior" message.
+    gdb_test "restart 3" \
+	     "\\\[Switching to inferior 1.*?main.*?$hello_c.*?alarm \\(240\\);"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 6"
+
+    gdb_test "restart 1" \
+	     "Switching to inferior 2.*?#0  main \\(\\) at.*?$goodbye_c.*mailand\\(\\);" \
+	     "restart 1 #2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 7"
+
+    gdb_test "checkpoint" "checkpoint 4: fork returned pid $::decimal.*" \
+	     "second checkpoint in inferior 2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  4 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 8"
+
+    gdb_test "checkpoint" "checkpoint 5: fork returned pid $::decimal.*" \
+	     "third checkpoint in inferior 2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  4 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 9"
+
+    gdb_test "continue" \
+	     "Inferior 2 \\(process $decimal\\) exited normally.*?Switching to $thr_or_proc.*?" \
+	     "continue to exit in checkpoint 1"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  4 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 5 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 10"
+
+    gdb_test "continue" \
+	     "Inferior 2 \\(process $decimal\\) exited normally.*?Switching to process $decimal.*?" \
+	     "continue to exit in checkpoint 5"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 4 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 11"
+
+    gdb_test "continue" \
+	     "Inferior 2 \\(process $decimal\\) exited normally.*?Switching to process $decimal.*?" \
+	     "continue to exit in checkpoint 4"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" ] \
+	    "info checkpoints 12"
+
+    gdb_test "checkpoint" "checkpoint 7: fork returned pid $::decimal.*" \
+	     "fourth checkpoint in inferior 2"
+
+    gdb_test "checkpoint" "checkpoint 8: fork returned pid $::decimal.*" \
+	     "fifth checkpoint in inferior 2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 6 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  7 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  8 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 13"
+
+    gdb_test "delete checkpoint 6" \
+	     "Please switch to another checkpoint before deleting the current one" \
+	     "delete checkpoint 6 #2"
+
+    gdb_test "restart 7" \
+	     "^Switching to process.*?#0  mailand \\(\\) at.*?$goodbye_c.*\}"
+
+    gdb_test "delete checkpoint 6" \
+	     "Killed process $::decimal"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 7 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  8 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 14"
+
+    gdb_test "delete checkpoint 8" \
+	     "Killed process $::decimal"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" ] \
+	    "info checkpoints 15"
+
+    gdb_test "checkpoint" "checkpoint 10: fork returned pid $::decimal.*" \
+	     "sixth checkpoint in inferior 2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 9 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 16"
+
+    gdb_test "inferior 1" "Switching to inferior 1.*?alarm \\(240\\);" \
+	     "inferior 1 #2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "  2 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\* 3 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 9 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 17"
+
+    gdb_test "kill" "\\\[Inferior 1 \\(process $::decimal\\) killed\\\]" \
+	     "kill inferior 1" \
+	     "Kill the program being debugged.*y or n. $" "y"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 9 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 18"
+
+    gdb_test "checkpoint" "The program is not being run\\." \
+	     "checkpoint in non-running inferior"
+
+    gdb_test "start" "Starting program.*?hello.*?alarm \\(240\\);"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 9 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 19"
+
+    gdb_test "checkpoint" "checkpoint 12: fork returned pid $::decimal.*" \
+	     "second checkpoint in inferior 1"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\* 11 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  12 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 9 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 20"
+}
+
+with_test_prefix "three inferiors with checkpoints" {
+    start_2_inferiors_catchpoint_on_inf_2
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?"] \
+	    "info checkpoints 1"
+
+    # Add a third inferior and exec into it.
+    gdb_test "add-inferior -exec $::binfile3" \
+	     "Added inferior 3.*" \
+	     "add inferior 3 with -exec $::exec3"
+
+    # Check that we have three inferiors.
+    gdb_test "info inferiors" \
+	     "Executable.*?\\* 1 .*?$::exec1.*? 2 .*?$::exec2.*? 3 .*?$::exec3.*?" \
+	     "check for three inferiors"
+
+    # Switch to inferior 3.
+    gdb_test "inferior 3" \
+	"Switching to inferior 3.*$::exec3.*"
+
+    # Start inferior 2:
+    if {[gdb_start_cmd] < 0} {
+	fail "start third inferior"
+    } else {
+	gdb_test "" "main.*" "start third inferior"
+    }
+
+    gdb_test "checkpoint" "checkpoint 3: fork returned pid $::decimal.*" \
+	     "first checkpoint in inferior 3"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 2 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 2"
+
+    gdb_test "inferior 1" "Switching to inferior 1.*?alarm \\(240\\);" \
+	     "inferior 1 #2"
+
+    gdb_test "checkpoint" "checkpoint 5: fork returned pid $::decimal.*" \
+	     "first checkpoint in inferior 1"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\* 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 3"
+
+    gdb_test "restart 1" \
+	     "Switching to inferior 2.*?#0  main \\(\\) at.*?$goodbye_c.*mailand\\(\\);" \
+	     "restart 1"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 4"
+
+    gdb_test "next" "foo\\(glob\\);"
+
+    gdb_test "checkpoint" "checkpoint 6: fork returned pid $::decimal.*" \
+	     "second checkpoint in inferior 2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 2 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 5"
+
+    gdb_test "inferior 3" "Switching to inferior 3.*?alarm \\(30\\);" \
+	     "inferior 3 #2"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 2 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  3 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 6"
+
+    gdb_test "kill" "\\\[Inferior 3 \\(process $::decimal\\) killed\\\]" \
+	     "kill inferior 3" \
+	     "Kill the program being debugged.*y or n. $" "y"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  0 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 1 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 7"
+
+    gdb_test "delete checkpoint 0" \
+	     "Killed process $::decimal"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "\\+ 1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "  6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 8"
+
+    gdb_test "restart 6" \
+	     "Switching to inferior 2.*?#0  main \\(\\) at.*?$goodbye_c.*foo\\(glob\\);"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 9"
+
+    gdb_test "inferior 3" "\\\[Switching to inferior 3 \\\[<null>\\\] \\(.*?$::exec3\\)\\\]" \
+	     "inferior 3 #3"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" ] \
+	    "info checkpoints 10"
+
+    gdb_test "start" "Starting program.*?hangout.*?alarm \\(30\\);"
+
+    gdb_test "checkpoint" "checkpoint 8: fork returned pid $::decimal.*" \
+	     "second checkpoint in inferior 3"
+
+    gdb_test "checkpoint" "checkpoint 9: fork returned pid $::decimal.*" \
+	     "third checkpoint in inferior 3"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  8 $thr_or_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 11"
+
+    gdb_test "delete checkpoint 8" \
+	     "Killed process $::decimal"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 12"
+
+    # Switch to inferior 1, add another checkpoint - so that there are
+    # three of them in inferior 1 - then switch back to inferior 1 and
+    # delete active checkpoint in inferior 1.  Then, switch to
+    # inferior 1 and attempt to add another checkpoint.  During
+    # development, a "Cannot access memory at address ..." message was
+    # seen.  This was a bug - there were several problems, but one
+    # of them was that the checkpoint in question was an "active"
+    # checkpoint.  The fix was to disallow this case.
+    gdb_test "inferior 1" "Switching to inferior 1.*?alarm \\(240\\);" \
+	     "inferior 1 #3"
+
+    gdb_test "checkpoint" "checkpoint 10: fork returned pid $::decimal.*" \
+	     "second checkpoint in inferior 1"
+
+    gdb_test "inferior 3" "Switching to inferior 3.*?alarm \\(30\\);" \
+	     "inferior 3 #4"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  5 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 13"
+
+    # Check that deleting active checkpoints in other (non-current)
+    # inferiors is disallowed.
+    gdb_test "delete checkpoint 4" \
+	     "Please switch to another checkpoint before deleting the current one\[ \r\n\]+in inferior 1"
+
+    # But deleting non-active checkpoints, even in other inferiors 
+    # should work.
+    gdb_test "delete checkpoint 5" \
+	     "Killed process $::decimal"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\+ 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\* 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 14"
+
+    gdb_test "inferior 1" "Switching to inferior 1.*?alarm \\(240\\);" \
+	     "inferior 1 #4"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\* 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 15"
+
+    gdb_test "checkpoint" "checkpoint 11: fork returned pid $::decimal.*" \
+	     "third checkpoint in inferior 1"
+
+    gdb_test "info checkpoints" \
+	[multi_line \
+	    "\\* 4 $thr_or_proc $main_proc at $::hex, file.*?$hello_c.*?" \
+	    "  10 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  11 $thr_or_proc at $::hex, file.*?$hello_c.*?" \
+	    "  1 $thr_or_proc $main_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 6 $thr_or_proc at $::hex, file.*?$goodbye_c.*?" \
+	    "\\+ 7 $thr_or_proc $main_proc at $::hex, file.*?$hangout_c.*?" \
+	    "  9 $thr_or_proc at $::hex, file.*?$hangout_c.*?" ] \
+	    "info checkpoints 16"
+
+    gdb_test "x/i \$pc" "=> $::hex <main.*"
+}