Skip to content

The ArrayIR class and YAKL Interoperation with Kokkos and other portability libraries

Jump to bottom
Matt Norman edited this page Nov 28, 2022 · 4 revisions

The ArrayIR class

Please see the following page for full API documentation of ArrayIR: https://mrnorman.github.io/yakl_api/html/namespaceArrayIR.html

In the file YAKL/external/ArrayIR.h, you'll find an ArrayIR namespace and class that have no dependencies on YAKL or any other portability library. It provides an "intermediate representation" of any portable library's multi-dimensional array object. It contains a data type and rank (both template parameters) as well as:

  1. A data pointer
  2. A set of dimension sizes represented as a std::array<size_t,rank> with the right-most varying the fastest (C-style)
  3. A memory type specifier that declares whether the data pointer is valid only on the host (ArrayIR::MEMORY_HOST), only on the device (ArrayIR::MEMORY_DEVICE), or on both the host and device (ArrayIR::MEMORY_SHARED)
  4. A char const * label used for debugging purposes if the user wants

You can create an ArrayIR object directly from a YAKL CArray or FArray object with the array object's create_ArrayIR() member function, and you can change the cv qualifiers if you want, e.g. my_obj.create_ArrayIR<float const>(), but that template parameter is optional and defaults to the type of the array object the method is called on. An FArray object will reverse its dimension sizes when creating an ArrayIR object to maintain the C-style expectations.

YAKL CArray and FArray objects can accept an ArrayIR object in their constructors. An FArray object will reverse the dimension sizes since it expects column-major indexing, and it will also accept an optional std::vector<int> parameter of lower bounds if you do not want the default of 1. There is also error checking to ensure you are not creating a memDevice Array of host only ArrayIR data or a memHost Array of device-only data.

ArrayIR objects are considered to be un-owned, and they do not manage or guarantee the validity or lifetimes of the data they point to. This is virtually impossible to guarantee across portability frameworks anyway. All data pointed to in an ArrayIR object is assumed to be contiguous in memory with no striding, and this corresponds to a YAKL Array object of styleC and a Kokkos View object of LayoutRight. All information must be provided to the ArrayIR constructor. Once an object is constructed, it cannot be modified (for safety), and only read-only accessors to information can be used afterward. The ArrayIR class may only be used from the host and not the device. It is only intended to hold array metadata to be transformed from one portability framework to another. Please create your portability framework's array object from the ArrayIR object, and then use that on the device. The ArrayIR class purposefully does not provide an overload of operator() because you should not be using it to access the data itself. It is only holding the array object's metadata.

It really is best not to try to use YAKL and another portability framework in the same CMake build directory (or to include YAKL and another portability framework in the same file). This is especially true with Kokkos because Kokkos and YAKL have different approaches to the build process for some backends. For instance, Kokkos wants to use the CXXFLAGS environment variable for files that use Kokkos, and with the CUDA backend, Kokkos wants to use nvcc_wrapper to compile all C++ files in the entire project. YAKL is a bit less invasive with respect to these issues. You provide the flags you want for YAKL files via the YAKL_[LANG]_FLAGS CMake variable, and they are only applied to C++ files actually using YAKL. Also, YAKL declared CUDA files as the CUDA language in CMake and lets CMake handle them accordingly. Older Kokkos versions require c++14 while newer versions require c++17 (the same is true for YAKL), and if that isn't compatible, the build will be difficult to manage.

The ArrayIR class, which has no c++ portability library dependencies allows users to pass Arrays between portability libraries without forcing them to directly interact in a build system.

API documentation:

Other interoperability issues

YAKL parallel_for launchers can use Kokkos View objects, and Kokkos parallel_for and parallel_reduce launchers can use YAKL Array objects.

You can use YAKL atomic functions inside Kokkos parallel_for launchers.

You can use YAKL SArray and FSArray objects inside Kokkos parallel_for launchers.

YAKL and Kokkos fence() operations are pretty much equivalent and can be used in either framework.

YAKL's YAKL_INLINE is similar to Kokkos's KOKKOS_INLINE_FUNCTION.

YAKL's YAKL_LAMBDA is similar to the Kokkos KOKKOS_LAMBDA

Clone this wiki locally