[gdb/python] Throw MemoryError in inferior.read_memory if malloc fails

Message ID 20240411105257.15421-1-tdevries@suse.de
State Committed
Headers
Series [gdb/python] Throw MemoryError in inferior.read_memory if malloc fails |

Checks

Context Check Description
linaro-tcwg-bot/tcwg_gdb_build--master-aarch64 success Testing passed
linaro-tcwg-bot/tcwg_gdb_build--master-arm success Testing passed
linaro-tcwg-bot/tcwg_gdb_check--master-arm fail Testing failed
linaro-tcwg-bot/tcwg_gdb_check--master-aarch64 success Testing passed

Commit Message

Tom de Vries April 11, 2024, 10:52 a.m. UTC
  PR python/31631 reports a gdb internal error when doing:
...
(gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
utils.c:709: internal-error: virtual memory exhausted.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
...

Fix this by throwing a python MemoryError, such that we have instead:
...
(gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
Python Exception <class 'MemoryError'>:
Error occurred in Python.
(gdb)
...

Likewise for DAP.

Tested on x86_64-linux.

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=31631
---
 gdb/python/lib/gdb/dap/memory.py         |  9 ++++++++-
 gdb/python/py-inferior.c                 | 11 ++++++++++-
 gdb/testsuite/gdb.dap/memory.exp         |  5 +++++
 gdb/testsuite/gdb.python/py-inferior.exp |  6 ++++++
 4 files changed, 29 insertions(+), 2 deletions(-)


base-commit: af925905211930677751678183f43c1bda13e013
  

Comments

Tom Tromey April 11, 2024, 4:07 p.m. UTC | #1
>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:

Tom> PR python/31631 reports a gdb internal error when doing:
Tom> ...
Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
Tom> utils.c:709: internal-error: virtual memory exhausted.
Tom> A problem internal to GDB has been detected,
Tom> further debugging may prove unreliable.
Tom> ...

Tom> Fix this by throwing a python MemoryError, such that we have instead:
Tom> ...
Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
Tom> Python Exception <class 'MemoryError'>:
Tom> Error occurred in Python.
Tom> (gdb)
Tom> ...

I tend to think you will regret opening this door, because I imagine
there are a large number of ways to crash gdb by passing nonsensical
values to Python APIs.

Tom>  @request("readMemory")
Tom>  @capability("supportsReadMemoryRequest")
Tom>  def read_memory(*, memoryReference: str, offset: int = 0, count: int, **extra):
Tom>      addr = int(memoryReference, 0) + offset
Tom> -    buf = gdb.selected_inferior().read_memory(addr, count)
Tom> +    oom = False
Tom> +    try:
Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
Tom> +    except MemoryError:
Tom> +        oom = True
Tom> +    if oom:
Tom> +        raise DAPException("Out of memory")

This should probably chain the memory error in the except block and
re-throw.  See https://peps.python.org/pep-3134/

However I don't really understand why this is needed.  Isn't the
exception already propagated back to the server thread?

Tom> +      /* We used to use xmalloc, which does this trick to avoid malloc
Tom> +	 returning a nullptr for a valid reason.  Keep doing the same.  */
Tom> +      if (length == 0)
Tom> +	length = 1;

This is most likely a workaround for vendor implementations of malloc
that return NULL for malloc(0).  See
https://www.gnu.org/software/gnulib/manual/html_node/malloc.html

However, here this is not necessary, because a 0-length memory read is
meaningless, and so this case can simply be reported as an error.

Tom
  
Tom de Vries April 12, 2024, 7:09 a.m. UTC | #2
On 4/11/24 18:07, Tom Tromey wrote:
>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
> 
> Tom> PR python/31631 reports a gdb internal error when doing:
> Tom> ...
> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
> Tom> utils.c:709: internal-error: virtual memory exhausted.
> Tom> A problem internal to GDB has been detected,
> Tom> further debugging may prove unreliable.
> Tom> ...
> 
> Tom> Fix this by throwing a python MemoryError, such that we have instead:
> Tom> ...
> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
> Tom> Python Exception <class 'MemoryError'>:
> Tom> Error occurred in Python.
> Tom> (gdb)
> Tom> ...
> 
> I tend to think you will regret opening this door, because I imagine
> there are a large number of ways to crash gdb by passing nonsensical
> values to Python APIs.
> 

Hm, ok then let's hope I don't regret it.

> Tom>  @request("readMemory")
> Tom>  @capability("supportsReadMemoryRequest")
> Tom>  def read_memory(*, memoryReference: str, offset: int = 0, count: int, **extra):
> Tom>      addr = int(memoryReference, 0) + offset
> Tom> -    buf = gdb.selected_inferior().read_memory(addr, count)
> Tom> +    oom = False
> Tom> +    try:
> Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
> Tom> +    except MemoryError:
> Tom> +        oom = True
> Tom> +    if oom:
> Tom> +        raise DAPException("Out of memory")
> 
> This should probably chain the memory error in the except block and
> re-throw.  See https://peps.python.org/pep-3134/
> 

Ack, that's what I had initially, but I ran into an error that I can no 
longer reproduce ... so, I'm not sure what happened there.  Anyway, fixed.

> However I don't really understand why this is needed.  Isn't the
> exception already propagated back to the server thread?
> 

Without it I run into:
...
FAIL: gdb.dap/memory.exp: exceptions in log file
...
because in the dap log we have:
...
READ: <<<{"seq": 7, "type": "request", "command": "readMemory", 
"arguments": {"memoryReference": "0x402010", "count": 
18446744073709551615}}>>>
Traceback (most recent call last):
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 157, in _handle_command
     body = _commands[params["command"]](**args)
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 300, in check
     return func(*args, **kwargs)
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 360, in sync_call
     return send_gdb_with_response(lambda: func(**args))
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 514, in send_gdb_with_response
     raise val
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 470, in __call__
     val = self.fn()
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", 
line 360, in <lambda>
     return send_gdb_with_response(lambda: func(**args))
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/startup.py", 
line 113, in ensure_gdb_thread
     return func(*args, **kwargs)
   File 
"/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/memory.py", 
line 28, in read_memory
     buf = gdb.selected_inferior().read_memory(addr, count)
MemoryError
WROTE: <<<{"request_seq": 7, "type": "response", "command": 
"readMemory", "success": false, "message": ""}>>>
...

So that needs fixing here:
...
diff --git a/gdb/python/lib/gdb/dap/server.py 
b/gdb/python/lib/gdb/dap/server.py
index 7eb87177710..8408232eceb 100644
--- a/gdb/python/lib/gdb/dap/server.py
+++ b/gdb/python/lib/gdb/dap/server.py
@@ -173,6 +173,9 @@ class Server:
              log_stack(LogLevel.FULL)
              result["success"] = False
              result["message"] = str(e)
+        except MemoryError as e:
+            result["success"] = False
+            result["message"] = str(e)
          except BaseException as e:
...

But I couldn't come up with a good rationale as to why I should handle 
MemoryError differently from BaseException, so I decided to catch it 
locally and turn it into a DAPException.

> Tom> +      /* We used to use xmalloc, which does this trick to avoid malloc
> Tom> +	 returning a nullptr for a valid reason.  Keep doing the same.  */
> Tom> +      if (length == 0)
> Tom> +	length = 1;
> 
> This is most likely a workaround for vendor implementations of malloc
> that return NULL for malloc(0).  See
> https://www.gnu.org/software/gnulib/manual/html_node/malloc.html
> 
> However, here this is not necessary, because a 0-length memory read is
> meaningless, and so this case can simply be reported as an error.

I went for backward compatibility here, but ok, fixed.  Updated version 
attached.

Thanks,
- Tom
  
Tom Tromey April 15, 2024, 4:16 p.m. UTC | #3
>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:

Tom> But I couldn't come up with a good rationale as to why I should handle
Tom> MemoryError differently from BaseException, so I decided to catch it
Tom> locally and turn it into a DAPException.

Yeah, makes sense.

Tom> +    try:
Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
Tom> +    except MemoryError:
Tom> +        raise DAPException("Out of memory")

Here I meant something like:

except MemoryError as e:
   raise DAPException(...) from e

There's an example of this in startup.py.

Tom
  
Tom de Vries April 16, 2024, 1:55 p.m. UTC | #4
On 4/15/24 18:16, Tom Tromey wrote:
>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
> 
> Tom> But I couldn't come up with a good rationale as to why I should handle
> Tom> MemoryError differently from BaseException, so I decided to catch it
> Tom> locally and turn it into a DAPException.
> 
> Yeah, makes sense.
> 
> Tom> +    try:
> Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
> Tom> +    except MemoryError:
> Tom> +        raise DAPException("Out of memory")
> 
> Here I meant something like:
> 
> except MemoryError as e:
>     raise DAPException(...) from e
> 
> There's an example of this in startup.py.

I see, I didn't know that possibility.

Anyway, AFAICT it doesn't seem to make a difference, it just makes the 
chaining explicit, which I prefer.

Pushed with that change.

Thanks,
- Tom
  
Simon Marchi April 16, 2024, 7:10 p.m. UTC | #5
On 4/12/24 3:09 AM, Tom de Vries wrote:
> On 4/11/24 18:07, Tom Tromey wrote:
>>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
>>
>> Tom> PR python/31631 reports a gdb internal error when doing:
>> Tom> ...
>> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
>> Tom> utils.c:709: internal-error: virtual memory exhausted.
>> Tom> A problem internal to GDB has been detected,
>> Tom> further debugging may prove unreliable.
>> Tom> ...
>>
>> Tom> Fix this by throwing a python MemoryError, such that we have instead:
>> Tom> ...
>> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
>> Tom> Python Exception <class 'MemoryError'>:
>> Tom> Error occurred in Python.
>> Tom> (gdb)
>> Tom> ...
>>
>> I tend to think you will regret opening this door, because I imagine
>> there are a large number of ways to crash gdb by passing nonsensical
>> values to Python APIs.
>>
> 
> Hm, ok then let's hope I don't regret it.
> 
>> Tom>  @request("readMemory")
>> Tom>  @capability("supportsReadMemoryRequest")
>> Tom>  def read_memory(*, memoryReference: str, offset: int = 0, count: int, **extra):
>> Tom>      addr = int(memoryReference, 0) + offset
>> Tom> -    buf = gdb.selected_inferior().read_memory(addr, count)
>> Tom> +    oom = False
>> Tom> +    try:
>> Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
>> Tom> +    except MemoryError:
>> Tom> +        oom = True
>> Tom> +    if oom:
>> Tom> +        raise DAPException("Out of memory")
>>
>> This should probably chain the memory error in the except block and
>> re-throw.  See https://peps.python.org/pep-3134/
>>
> 
> Ack, that's what I had initially, but I ran into an error that I can no longer reproduce ... so, I'm not sure what happened there.  Anyway, fixed.
> 
>> However I don't really understand why this is needed.  Isn't the
>> exception already propagated back to the server thread?
>>
> 
> Without it I run into:
> ...
> FAIL: gdb.dap/memory.exp: exceptions in log file
> ...
> because in the dap log we have:
> ...
> READ: <<<{"seq": 7, "type": "request", "command": "readMemory", "arguments": {"memoryReference": "0x402010", "count": 18446744073709551615}}>>>
> Traceback (most recent call last):
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 157, in _handle_command
>     body = _commands[params["command"]](**args)
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 300, in check
>     return func(*args, **kwargs)
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 360, in sync_call
>     return send_gdb_with_response(lambda: func(**args))
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 514, in send_gdb_with_response
>     raise val
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 470, in __call__
>     val = self.fn()
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 360, in <lambda>
>     return send_gdb_with_response(lambda: func(**args))
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/startup.py", line 113, in ensure_gdb_thread
>     return func(*args, **kwargs)
>   File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/memory.py", line 28, in read_memory
>     buf = gdb.selected_inferior().read_memory(addr, count)
> MemoryError
> WROTE: <<<{"request_seq": 7, "type": "response", "command": "readMemory", "success": false, "message": ""}>>>
> ...
> 
> So that needs fixing here:
> ...
> diff --git a/gdb/python/lib/gdb/dap/server.py b/gdb/python/lib/gdb/dap/server.py
> index 7eb87177710..8408232eceb 100644
> --- a/gdb/python/lib/gdb/dap/server.py
> +++ b/gdb/python/lib/gdb/dap/server.py
> @@ -173,6 +173,9 @@ class Server:
>              log_stack(LogLevel.FULL)
>              result["success"] = False
>              result["message"] = str(e)
> +        except MemoryError as e:
> +            result["success"] = False
> +            result["message"] = str(e)
>          except BaseException as e:
> ...
> 
> But I couldn't come up with a good rationale as to why I should handle MemoryError differently from BaseException, so I decided to catch it locally and turn it into a DAPException.
> 
>> Tom> +      /* We used to use xmalloc, which does this trick to avoid malloc
>> Tom> +     returning a nullptr for a valid reason.  Keep doing the same.  */
>> Tom> +      if (length == 0)
>> Tom> +    length = 1;
>>
>> This is most likely a workaround for vendor implementations of malloc
>> that return NULL for malloc(0).  See
>> https://www.gnu.org/software/gnulib/manual/html_node/malloc.html
>>
>> However, here this is not necessary, because a 0-length memory read is
>> meaningless, and so this case can simply be reported as an error.
> 
> I went for backward compatibility here, but ok, fixed.  Updated version attached.
> 
> Thanks,
> - Tom

In an Asan-enabled build, I get:


 74 (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)^M
 75 =================================================================^M
 76 ^[[1m^[[31m==791047==ERROR: AddressSanitizer: requested allocation size 0xffffffffffffffff (0x800 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x100    00000000 (thread T0)^M
 77 ^[[1m^[[0m    #0 0x7f02c58d49cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69^M
 78     #1 0x55cc5937f257 in infpy_read_memory /home/smarchi/src/binutils-gdb/gdb/python/py-inferior.c:565^M
 79     #2 0x7f02c4ea9709 in method_vectorcall_VARARGS_KEYWORDS ../Objects/descrobject.c:364^M
 80 ^M
 81 ==791047==HINT: if you don't care about these errors you may set allocator_may_return_null=1^M
 82 SUMMARY: AddressSanitizer: allocation-size-too-big ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 in __interceptor_malloc^M

Simon
  
Tom de Vries April 17, 2024, 8:29 a.m. UTC | #6
On 4/16/24 21:10, Simon Marchi wrote:
> On 4/12/24 3:09 AM, Tom de Vries wrote:
>> On 4/11/24 18:07, Tom Tromey wrote:
>>>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
>>>
>>> Tom> PR python/31631 reports a gdb internal error when doing:
>>> Tom> ...
>>> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
>>> Tom> utils.c:709: internal-error: virtual memory exhausted.
>>> Tom> A problem internal to GDB has been detected,
>>> Tom> further debugging may prove unreliable.
>>> Tom> ...
>>>
>>> Tom> Fix this by throwing a python MemoryError, such that we have instead:
>>> Tom> ...
>>> Tom> (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)
>>> Tom> Python Exception <class 'MemoryError'>:
>>> Tom> Error occurred in Python.
>>> Tom> (gdb)
>>> Tom> ...
>>>
>>> I tend to think you will regret opening this door, because I imagine
>>> there are a large number of ways to crash gdb by passing nonsensical
>>> values to Python APIs.
>>>
>>
>> Hm, ok then let's hope I don't regret it.
>>
>>> Tom>  @request("readMemory")
>>> Tom>  @capability("supportsReadMemoryRequest")
>>> Tom>  def read_memory(*, memoryReference: str, offset: int = 0, count: int, **extra):
>>> Tom>      addr = int(memoryReference, 0) + offset
>>> Tom> -    buf = gdb.selected_inferior().read_memory(addr, count)
>>> Tom> +    oom = False
>>> Tom> +    try:
>>> Tom> +        buf = gdb.selected_inferior().read_memory(addr, count)
>>> Tom> +    except MemoryError:
>>> Tom> +        oom = True
>>> Tom> +    if oom:
>>> Tom> +        raise DAPException("Out of memory")
>>>
>>> This should probably chain the memory error in the except block and
>>> re-throw.  See https://peps.python.org/pep-3134/
>>>
>>
>> Ack, that's what I had initially, but I ran into an error that I can no longer reproduce ... so, I'm not sure what happened there.  Anyway, fixed.
>>
>>> However I don't really understand why this is needed.  Isn't the
>>> exception already propagated back to the server thread?
>>>
>>
>> Without it I run into:
>> ...
>> FAIL: gdb.dap/memory.exp: exceptions in log file
>> ...
>> because in the dap log we have:
>> ...
>> READ: <<<{"seq": 7, "type": "request", "command": "readMemory", "arguments": {"memoryReference": "0x402010", "count": 18446744073709551615}}>>>
>> Traceback (most recent call last):
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 157, in _handle_command
>>      body = _commands[params["command"]](**args)
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 300, in check
>>      return func(*args, **kwargs)
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 360, in sync_call
>>      return send_gdb_with_response(lambda: func(**args))
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 514, in send_gdb_with_response
>>      raise val
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 470, in __call__
>>      val = self.fn()
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/server.py", line 360, in <lambda>
>>      return send_gdb_with_response(lambda: func(**args))
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/startup.py", line 113, in ensure_gdb_thread
>>      return func(*args, **kwargs)
>>    File "/data/vries/gdb/leap-15-5/build/gdb/data-directory/python/gdb/dap/memory.py", line 28, in read_memory
>>      buf = gdb.selected_inferior().read_memory(addr, count)
>> MemoryError
>> WROTE: <<<{"request_seq": 7, "type": "response", "command": "readMemory", "success": false, "message": ""}>>>
>> ...
>>
>> So that needs fixing here:
>> ...
>> diff --git a/gdb/python/lib/gdb/dap/server.py b/gdb/python/lib/gdb/dap/server.py
>> index 7eb87177710..8408232eceb 100644
>> --- a/gdb/python/lib/gdb/dap/server.py
>> +++ b/gdb/python/lib/gdb/dap/server.py
>> @@ -173,6 +173,9 @@ class Server:
>>               log_stack(LogLevel.FULL)
>>               result["success"] = False
>>               result["message"] = str(e)
>> +        except MemoryError as e:
>> +            result["success"] = False
>> +            result["message"] = str(e)
>>           except BaseException as e:
>> ...
>>
>> But I couldn't come up with a good rationale as to why I should handle MemoryError differently from BaseException, so I decided to catch it locally and turn it into a DAPException.
>>
>>> Tom> +      /* We used to use xmalloc, which does this trick to avoid malloc
>>> Tom> +     returning a nullptr for a valid reason.  Keep doing the same.  */
>>> Tom> +      if (length == 0)
>>> Tom> +    length = 1;
>>>
>>> This is most likely a workaround for vendor implementations of malloc
>>> that return NULL for malloc(0).  See
>>> https://www.gnu.org/software/gnulib/manual/html_node/malloc.html
>>>
>>> However, here this is not necessary, because a 0-length memory read is
>>> meaningless, and so this case can simply be reported as an error.
>>
>> I went for backward compatibility here, but ok, fixed.  Updated version attached.
>>
>> Thanks,
>> - Tom
> 
> In an Asan-enabled build, I get:
> 
> 
>   74 (gdb) python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)^M
>   75 =================================================================^M
>   76 ^[[1m^[[31m==791047==ERROR: AddressSanitizer: requested allocation size 0xffffffffffffffff (0x800 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x100    00000000 (thread T0)^M
>   77 ^[[1m^[[0m    #0 0x7f02c58d49cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69^M
>   78     #1 0x55cc5937f257 in infpy_read_memory /home/smarchi/src/binutils-gdb/gdb/python/py-inferior.c:565^M
>   79     #2 0x7f02c4ea9709 in method_vectorcall_VARARGS_KEYWORDS ../Objects/descrobject.c:364^M
>   80 ^M
>   81 ==791047==HINT: if you don't care about these errors you may set allocator_may_return_null=1^M
>   82 SUMMARY: AddressSanitizer: allocation-size-too-big ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 in __interceptor_malloc^M
> 

Hi Simon,

thanks for the report.

I've submitted a fix here ( 
https://sourceware.org/pipermail/gdb-patches/2024-April/208178.html ).

Thanks,
- Tom
  

Patch

diff --git a/gdb/python/lib/gdb/dap/memory.py b/gdb/python/lib/gdb/dap/memory.py
index dd62b0e7ba6..0fd5c2128e5 100644
--- a/gdb/python/lib/gdb/dap/memory.py
+++ b/gdb/python/lib/gdb/dap/memory.py
@@ -18,13 +18,20 @@  import base64
 import gdb
 
 from .server import capability, request
+from .startup import DAPException
 
 
 @request("readMemory")
 @capability("supportsReadMemoryRequest")
 def read_memory(*, memoryReference: str, offset: int = 0, count: int, **extra):
     addr = int(memoryReference, 0) + offset
-    buf = gdb.selected_inferior().read_memory(addr, count)
+    oom = False
+    try:
+        buf = gdb.selected_inferior().read_memory(addr, count)
+    except MemoryError:
+        oom = True
+    if oom:
+        raise DAPException("Out of memory")
     return {
         "address": hex(addr),
         "data": base64.b64encode(buf).decode("ASCII"),
diff --git a/gdb/python/py-inferior.c b/gdb/python/py-inferior.c
index 795ac655ddd..e825e649533 100644
--- a/gdb/python/py-inferior.c
+++ b/gdb/python/py-inferior.c
@@ -562,7 +562,16 @@  infpy_read_memory (PyObject *self, PyObject *args, PyObject *kw)
       scoped_restore_current_inferior_for_memory restore_inferior
 	(inf->inferior);
 
-      buffer.reset ((gdb_byte *) xmalloc (length));
+      /* We used to use xmalloc, which does this trick to avoid malloc
+	 returning a nullptr for a valid reason.  Keep doing the same.  */
+      if (length == 0)
+	length = 1;
+
+      void *p = malloc (length);
+      if (p == nullptr)
+	return PyErr_NoMemory ();
+
+      buffer.reset ((gdb_byte *) p);
 
       read_memory (addr, buffer.get (), length);
     }
diff --git a/gdb/testsuite/gdb.dap/memory.exp b/gdb/testsuite/gdb.dap/memory.exp
index 2e911f4dc77..4e2e361289a 100644
--- a/gdb/testsuite/gdb.dap/memory.exp
+++ b/gdb/testsuite/gdb.dap/memory.exp
@@ -55,6 +55,11 @@  set obj [dap_check_request_and_response "evaluate global pointer" \
 	     evaluate {o expression [s thirty_two_p]}]
 set addr [dict get [lindex $obj 0] body memoryReference]
 
+set obj [dap_request_and_response \
+	     readMemory [format {o memoryReference [s %s] count [i 18446744073709551615]} $addr]]
+set response [lindex $obj 0]
+gdb_assert { [dict get $response success] == "false" } "read memory, count to big"
+
 set obj [dap_check_request_and_response "read memory" \
 	     readMemory [format {o memoryReference [s %s] count [i 4]} $addr]]
 
diff --git a/gdb/testsuite/gdb.python/py-inferior.exp b/gdb/testsuite/gdb.python/py-inferior.exp
index c14f2d2796c..4c19e259159 100644
--- a/gdb/testsuite/gdb.python/py-inferior.exp
+++ b/gdb/testsuite/gdb.python/py-inferior.exp
@@ -34,6 +34,12 @@  switch [get_endianness] {
     big { set python_pack_char ">" }
 }
 
+gdb_test \
+    "python gdb.selected_inferior().read_memory (0, 0xffffffffffffffff)" \
+    [multi_line \
+	 [string_to_regexp "Python Exception <class 'MemoryError'>: "] \
+	 [string_to_regexp "Error occurred in Python."]]
+
 # Test memory read operations without execution.
 
 gdb_py_test_silent_cmd "python addr = gdb.lookup_global_symbol ('int8_global').value().address" \