[2/5,gfortran] Translate allocate directive (OpenMP 5.0).

  gcc/fortran/ChangeLog:

	* trans-openmp.c (gfc_trans_omp_clauses): Handle OMP_LIST_ALLOCATOR.
	(gfc_trans_omp_allocate): New function.
	(gfc_trans_omp_directive): Handle EXEC_OMP_ALLOCATE.

gcc/ChangeLog:

	* tree-pretty-print.c (dump_omp_clause): Handle OMP_CLAUSE_ALLOCATOR.
	(dump_generic_node): Handle OMP_ALLOCATE.
	* tree.def (OMP_ALLOCATE): New.
	* tree.h (OMP_ALLOCATE_CLAUSES): Likewise.
	(OMP_ALLOCATE_DECL): Likewise.
	(OMP_ALLOCATE_ALLOCATOR): Likewise.
	* tree.c (omp_clause_num_ops): Add entry for OMP_CLAUSE_ALLOCATOR.

gcc/testsuite/ChangeLog:

	* gfortran.dg/gomp/allocate-6.f90: New test.
---
 gcc/fortran/trans-openmp.c                    | 44 ++++++++++++
 gcc/testsuite/gfortran.dg/gomp/allocate-6.f90 | 72 +++++++++++++++++++
 gcc/tree-core.h                               |  3 +
 gcc/tree-pretty-print.c                       | 19 +++++
 gcc/tree.c                                    |  1 +
 gcc/tree.def                                  |  4 ++
 gcc/tree.h                                    | 11 +++
 7 files changed, 154 insertions(+)
 create mode 100644 gcc/testsuite/gfortran.dg/gomp/allocate-6.f90
  

On Thu, Jan 13, 2022 at 02:53:17PM +0000, Hafiz Abid Qadeer wrote:
> gcc/fortran/ChangeLog:
> 
> 	* trans-openmp.c (gfc_trans_omp_clauses): Handle OMP_LIST_ALLOCATOR.
> 	(gfc_trans_omp_allocate): New function.
> 	(gfc_trans_omp_directive): Handle EXEC_OMP_ALLOCATE.
> 
> gcc/ChangeLog:
> 
> 	* tree-pretty-print.c (dump_omp_clause): Handle OMP_CLAUSE_ALLOCATOR.
> 	(dump_generic_node): Handle OMP_ALLOCATE.
> 	* tree.def (OMP_ALLOCATE): New.
> 	* tree.h (OMP_ALLOCATE_CLAUSES): Likewise.
> 	(OMP_ALLOCATE_DECL): Likewise.
> 	(OMP_ALLOCATE_ALLOCATOR): Likewise.
> 	* tree.c (omp_clause_num_ops): Add entry for OMP_CLAUSE_ALLOCATOR.
> 
> gcc/testsuite/ChangeLog:
> 
> 	* gfortran.dg/gomp/allocate-6.f90: New test.

There is another issue besides what I wrote in my last review,
and I'm afraid I don't know what to do about it, hoping Tobias
has some ideas.
The problem is that without the allocate-stmt associated allocate directive,
Fortran allocatables are easily always allocated with malloc and freed with
free.  The deallocation can be implicit through reallocation, or explicit
deallocate statement etc.
But when some allocatables are now allocated with a different
allocator (when allocate-stmt associated allocate directive is used),
some allocatables are allocated with malloc and others with GOMP_alloc
but we need to free them with the corresponding allocator based on how
they were allocated, what has been allocated with malloc should be
deallocated with free, what has been allocated with GOMP_alloc should be
deallocated with GOMP_free.
The deallocation can be done in a completely different TU from where it has
been allocated, in theory it could be also not compiled with -fopenmp, etc.
So, I'm afraid we need to store somewhere whether we used malloc or
GOMP_alloc for the allocation (say somewhere in the array descriptor and for
other stuff somewhere on the side?) and slow down all code that needs
deallocation to check that bit (or say we don't support
deallocation/reallocation of OpenMP allocated allocatables without -fopenmp
on the deallocation TU and only slow down -fopenmp compiled code)?

Tobias, thoughts on this?

	Jakub
  

Hi Jakub,

On 11.10.22 14:24, Jakub Jelinek wrote:

There is another issue besides what I wrote in my last review,
and I'm afraid I don't know what to do about it, hoping Tobias
has some ideas.
The problem is that without the allocate-stmt associated allocate directive,
Fortran allocatables are easily always allocated with malloc and freed with
free.  The deallocation can be implicit through reallocation, or explicit
deallocate statement etc.
...
But when some allocatables are now allocated with a different
allocator (when allocate-stmt associated allocate directive is used),
some allocatables are allocated with malloc and others with GOMP_alloc
but we need to free them with the corresponding allocator based on how
they were allocated, what has been allocated with malloc should be
deallocated with free, what has been allocated with GOMP_alloc should be
deallocated with GOMP_free.



I think the most common case is:

integer, allocatable :: var(:)
!$omp allocators allocator(my_alloc) ! must be in same scope as decl of 'var'
...
! optionally: deallocate(var)
end ! of scope: block/subroutine/... - automatic deallocation

Those can be easily handled. It gets more complicated with control flow:

if (...) then
  !$omp allocators allocator(...)
  allocate(...)
else
  allocate (...)
endif



However, the problem is really that there is is no mandatory
'!$omp deallocators' and also the wording like:

"If any operation of the base language causes a reallocation of
an array that is allocated with a memory allocator then that
memory allocator will be used to release the current memory
and to allocate the new memory." (OpenMP 5.0 wording)

There has been some attempt to relax the rules a bit, e.g. by
adding the wording:
"For allocated allocatable components of such variables, the allocator that
will be used for the deallocation and allocation is unspecified."

And some wording change (→issues 3189) to clarify related component issues.

But nonetheless, there is still the issue of:

(a) explicit DEALLOCATE in some other translation unit
(b) some intrinsic operation which reallocate the memory, either via libgomp
or in the source code:
  a = [1,2,3]  ! possibly reallocates
  str = trim(str) ! possibly reallocates
where the first one calls 'realloc' directly in the code and the second one
calls 'libgomp' for that.

 * * *

I don't see a good solution – and there is in principle the same issue with
unified-shared memory (USM) on hardware that does not support transparently
accessing all host memory on the device.

Compilers support this case by allocating memory in some special memory,
which is either accessible from both sides ('pinned') or migrates on the
first access from the device side - but remains there until the accessing
device kernel ends ('managed memory').

Newer hardware (+ associated Linux kernel support) permit accessing all
memory in a somewhat fast way, avoiding this issue (and special handling
is then left to the user.) For AMDGCN, my understanding is that all hardware
supported by GCC supports this - but glacial speed until the last hardware
architectures. For Nvidia, this is supported since Pascal (I think for Titan X,
P100, i.e. sm_5.2/sm_60) - but I believe not for all Pascal/Kepler hardware.

I mention this because the USM implementation at
https://gcc.gnu.org/pipermail/gcc-patches/2022-July/597976.html
suffers from this.
And https://gcc.gnu.org/pipermail/gcc-patches/2022-September/601059.html
tries to solve the the 'trim' example issue above - i.e. the case where
libgomp reallocates pinned/managed (pseudo-)USM memory.

 * * *

The deallocation can be done in a completely different TU from where it has
been allocated, in theory it could be also not compiled with -fopenmp, etc.
So, I'm afraid we need to store somewhere whether we used malloc or
GOMP_alloc for the allocation (say somewhere in the array descriptor and for
other stuff somewhere on the side?) and slow down all code that needs
deallocation to check that bit (or say we don't support
deallocation/reallocation of OpenMP allocated allocatables without -fopenmp
on the deallocation TU and only slow down -fopenmp compiled code)?

The problem with storing is that gfortran inserts the malloc/realloc/free calls directly, i.e. without library preloading, intercepting those libcalls, I do not see how it can work at all.

I also do not know how to handle the pinned-memory case above correctly, either.

One partial support would be requiring that the code using allocatables cannot do any reallocation/deallocation by only permitting calls to procedures which do not permit allocatables. (Such that no reallocation can happen.) – And print a 'sorry' for the rest.

Other implementations seem to have a Fortran library call for (re)allocations, which permits to swap the allocator from the generic one to the omp_default_mem_alloc.

* * *

In terms of the array descriptor, we have inside 'struct dtype_type'  the 'signed short attribute', which currently only holds CFI_attribute_pointer/CFI_attribute_allocatable/CFI_attribute_other (=0,1,2). And this is only used together with ISO C binding, permitting to use the other bits for other purpose (for the non-ISO-C case). Still, the question is *how* to use it in that case.

Thoughts on the generic issue on those thoughts?

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
  

On Tue, Oct 11, 2022 at 03:22:02PM +0200, Tobias Burnus wrote:
> Hi Jakub,
> 
> On 11.10.22 14:24, Jakub Jelinek wrote:
> 
> There is another issue besides what I wrote in my last review,
> and I'm afraid I don't know what to do about it, hoping Tobias
> has some ideas.
> The problem is that without the allocate-stmt associated allocate directive,
> Fortran allocatables are easily always allocated with malloc and freed with
> free.  The deallocation can be implicit through reallocation, or explicit
> deallocate statement etc.
> ...
> But when some allocatables are now allocated with a different
> allocator (when allocate-stmt associated allocate directive is used),
> some allocatables are allocated with malloc and others with GOMP_alloc
> but we need to free them with the corresponding allocator based on how
> they were allocated, what has been allocated with malloc should be
> deallocated with free, what has been allocated with GOMP_alloc should be
> deallocated with GOMP_free.
> 
> 
> 
> I think the most common case is:
> 
> integer, allocatable :: var(:)
> !$omp allocators allocator(my_alloc) ! must be in same scope as decl of 'var'
> ...
> ! optionally: deallocate(var)
> end ! of scope: block/subroutine/... - automatic deallocation

So you talk here about the declarative directive the patch does sorry on,
or about the executable one above allocate stmt?

Anyway, even this simple case has the problem that one can have
subroutine foo (var)
  integer, allocatable:: var(:)
  var = [1, 2, 3] ! reallocate
end subroutine
and call foo (var) above.

> Those can be easily handled. It gets more complicated with control flow:
> 
> if (...) then
>  !$omp allocators allocator(...)
>  allocate(...)
> else
>  allocate (...)
> endif
> 
> 
> 
> However, the problem is really that there is is no mandatory
> '!$omp deallocators' and also the wording like:
> 
> "If any operation of the base language causes a reallocation of
> an array that is allocated with a memory allocator then that
> memory allocator will be used to release the current memory
> and to allocate the new memory." (OpenMP 5.0 wording)
> 
> There has been some attempt to relax the rules a bit, e.g. by
> adding the wording:
> "For allocated allocatable components of such variables, the allocator that
> will be used for the deallocation and allocation is unspecified."
> 
> And some wording change (→issues 3189) to clarify related component issues.
> 
> But nonetheless, there is still the issue of:
> 
> (a) explicit DEALLOCATE in some other translation unit
> (b) some intrinsic operation which reallocate the memory, either via libgomp
> or in the source code:
>  a = [1,2,3]  ! possibly reallocates
>  str = trim(str) ! possibly reallocates
> where the first one calls 'realloc' directly in the code and the second one
> calls 'libgomp' for that.
> 
> * * *
> 
> I don't see a good solution – and there is in principle the same issue with
> unified-shared memory (USM) on hardware that does not support transparently
> accessing all host memory on the device.
> 
> Compilers support this case by allocating memory in some special memory,
> which is either accessible from both sides ('pinned') or migrates on the
> first access from the device side - but remains there until the accessing
> device kernel ends ('managed memory').
> 
> Newer hardware (+ associated Linux kernel support) permit accessing all
> memory in a somewhat fast way, avoiding this issue (and special handling
> is then left to the user.) For AMDGCN, my understanding is that all hardware
> supported by GCC supports this - but glacial speed until the last hardware
> architectures. For Nvidia, this is supported since Pascal (I think for Titan X,
> P100, i.e. sm_5.2/sm_60) - but I believe not for all Pascal/Kepler hardware.
> 
> I mention this because the USM implementation at
> https://gcc.gnu.org/pipermail/gcc-patches/2022-July/597976.html
> suffers from this.
> And https://gcc.gnu.org/pipermail/gcc-patches/2022-September/601059.html
> tries to solve the the 'trim' example issue above - i.e. the case where
> libgomp reallocates pinned/managed (pseudo-)USM memory.
> 
> * * *
> 
> The deallocation can be done in a completely different TU from where it has
> been allocated, in theory it could be also not compiled with -fopenmp, etc.
> So, I'm afraid we need to store somewhere whether we used malloc or
> GOMP_alloc for the allocation (say somewhere in the array descriptor and for
> other stuff somewhere on the side?) and slow down all code that needs
> deallocation to check that bit (or say we don't support
> deallocation/reallocation of OpenMP allocated allocatables without -fopenmp
> on the deallocation TU and only slow down -fopenmp compiled code)?
> 
> The problem with storing is that gfortran inserts the malloc/realloc/free calls directly, i.e. without library preloading, intercepting those libcalls, I do not see how it can work at all.

Well, it can use a weak symbol, if not linked against libgomp, the bit
that it is OpenMP shouldn't be set and so realloc/free will be used
and do
  if (arrdescr.gomp_alloced_bit)
    GOMP_free (arrdescr.data, 0);
  else
    free (arrdescr.data);
and similar.  And I think we can just document that we do this only for
-fopenmp compiled code.
But do we have a place to store that bit?  I presume in array descriptors
there could be some bit for it, but what to do about scalar allocatables,
or allocatable components etc.?
In theory we could use ugly stuff like if all the allocations would be
guaranteed to have at least 2 byte alignment use LSB bit of the pointer
to mark GOMP_alloc allocated memory for the scalar allocatables etc. but
then would need in -fopenmp compiled code to strip it away.

As for pinned memory, if it is allocated through libgomp allocators, that
should just work if GOMP_free/GOMP_realloc is used, that is why we have
those extra data in front of the allocations where we store everything we
need.  But those also make the OpenMP allocations incompatible with
malloc/free allocations.

	Jakub
  

On Tue, Oct 11, 2022 at 04:15:25PM +0200, Jakub Jelinek wrote:
> Well, it can use a weak symbol, if not linked against libgomp, the bit
> that it is OpenMP shouldn't be set and so realloc/free will be used
> and do
>   if (arrdescr.gomp_alloced_bit)
>     GOMP_free (arrdescr.data, 0);
>   else
>     free (arrdescr.data);
> and similar.  And I think we can just document that we do this only for
> -fopenmp compiled code.
> But do we have a place to store that bit?  I presume in array descriptors
> there could be some bit for it, but what to do about scalar allocatables,
> or allocatable components etc.?
> In theory we could use ugly stuff like if all the allocations would be
> guaranteed to have at least 2 byte alignment use LSB bit of the pointer
> to mark GOMP_alloc allocated memory for the scalar allocatables etc. but
> then would need in -fopenmp compiled code to strip it away.
> 
> As for pinned memory, if it is allocated through libgomp allocators, that
> should just work if GOMP_free/GOMP_realloc is used, that is why we have
> those extra data in front of the allocations where we store everything we
> need.  But those also make the OpenMP allocations incompatible with
> malloc/free allocations.

Yet another option would be to change the way our OpenMP allocators work,
instead of having allocation internal data before the allocated memory
have them somewhere on the side and use some data structures mapping
ranges of virtual memory to the allocation data.
We'd either need to use mmap to have better control on where exactly
we allocate stuff so that the on the side data structures wouldn't need
to be for every allocation, or do those for every allocation perhaps with
merging of adjacent allocations or something similar.
Disadvantage is that it would be slower and might need more locking etc.,
advantage is that it could be then malloc/free compatible, any not tracked
address would be forwarded from GOMP_free to free etc.  And we'd not waste
e.g. precious pinned etc. memory especially when doing allocations with very
high alignment, where the data before allocation means we can waste up to
max (32, alignment - 1) of extra memory.  And gfortran
inline emitted reallocation/deallocation could just emit GOMP_realloc/free
always for -fopenmp.  The way GOMP_ allocators are currently written, it is
our internal choice if we do it the current way or the on the side way or
some other way, but if we'd guarantee free compatibility we'd make it part
of the ABI.

CCing DJ and Carlos if they have thoughts about this.
The OpenMP spec essentially requires that allocations through its allocator
remember somewhere with which allocator (and its exact properties) each
allocation has been done, so that it can be taken into account during
reallocation or freeing.

	Jakub
  

On 11.10.22 16:15, Jakub Jelinek wrote:

I think the most common case is:

integer, allocatable :: var(:)
!$omp allocators allocator(my_alloc) ! must be in same scope as decl of 'var'
...
! optionally: deallocate(var)
end ! of scope: block/subroutine/... - automatic deallocation



So you talk here about the declarative directive the patch does sorry on,
or about the executable one above allocate stmt?

Here, I was only talking about the most common usage case, with the
assumption that the user code does not cause any reallocation.

I later talked about accepting only code which cannot cause
reallocation (compile-time check of the code contained in the
scope).

Thus, a 'call foo(a)' would be fine, but not for ...


Anyway, even this simple case has the problem that one can have
subroutine foo (var)
  integer, allocatable:: var(:)

a 'foo' that has an 'allocatable' attribute for the dummy argument.
I think in the common case, it has not – such that most code can run w/o running into this issue.

However, for code like
  type t
    real, allocatable :: x(:), y(:), z(:)
  end type t
  type(t) :: var
  !$omp allocators(my_alloc)
  allocate(var%x(N), var%y(N), var%z(N))

  call bar(var%x)
  call foo(var)

it is more difficult: 'bar' works (if its dummy argument is not 'allocatable')
but for 'foo', the (re|de)allocation cannot be ruled out.
Thus, we always have to 'sorry' for such a code – and I fear it could be somewhat
common.



Well, it can use a weak symbol, if not linked against libgomp, the bit
that it is OpenMP shouldn't be set and so realloc/free will be used
and do
  if (arrdescr.gomp_alloced_bit)
    GOMP_free (arrdescr.data, 0);
  else
    free (arrdescr.data);
and similar.  And I think we can just document that we do this only for
-fopenmp compiled code.
But do we have a place to store that bit?

I presume in array descriptors
there could be some bit for it, but what to do about scalar allocatables,
or allocatable components etc.?

As mentioned, we could use the 'dtype.attribute' field which is currently not really used – and if, only 2 of the 16 bits are used. But you are right that for scalar allocatables, we do not use array descriptors (except with BIND(C)). Hmm.

For allocatable components, the same applied: If arrays, then there is an array descriptor – for scalars, there isn't. (And storing the length of a scalar character string with deferred length uses an aux variable + has lots of bugs.)

In theory we could use ugly stuff like if all the allocations would be
guaranteed to have at least 2 byte alignment use LSB bit of the pointer
to mark GOMP_alloc allocated memory for the scalar allocatables etc. but
then would need in -fopenmp compiled code to strip it away.

I think we could do tricks with scalar allocatable variable – but it will be more complicated with scalar allocatable components. Hmm.

As for pinned memory, if it is allocated through libgomp allocators, that
should just work if GOMP_free/GOMP_realloc is used, that is why we have
those extra data in front of the allocations where we store everything we
need.  But those also make the OpenMP allocations incompatible with
malloc/free allocations.


The problem of making pseudo-USM work is that it has to be applied to all (stack,heap) memory – which implies that all code using malloc/free needs to be either call the GOMP version or the GLIBC version, but shall not mix one or the other. – Thus, calling some library or any other file that was not compiled with -f... will have issues with malloc/free. Another issue is that variables not allocated via GOMP_* will not be accessible on the device in that case.

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955