From 31773a2f0d30f3f64c82939f60fc4da32cf33261 Mon Sep 17 00:00:00 2001 From: Alan Dayton <6393677+adayton1@users.noreply.github.com> Date: Tue, 5 Mar 2024 09:34:32 -0800 Subject: [PATCH] V2024.02.0 RC (#252) * Depend on develop version of TPLs * Tweak version constraints * Update Umpire package * Fix syntax error in Umpire package * Update packages for CHAI and Umpire * Depend on newer camp * Update versions for RAJA and Umpire * Fixes for libcpp builds * Add SLURM_OVERLAP variable * Add greater control for when CHAI is a submodule * Fix hip build with resource manager disabled * Depend on camp@main * Missed one reference to camp@master * Bug fixes for HIP and managed_ptr * Clean up managed_ptr.hpp * Fix host build warning * Customize block size for HIP build * Add a managed_ptr example * Fix HIP kernel launches * Add reproducer for HIP managed_ptr errors * Clean up managed_ptr reproducer * Add a simpler reproducer * Add simplest reproducer * Fix macro * Fix mutex lock in evict() that overlaps with lock in free() * initial changes to add CHAI_ cmake prefix * fixing cmake option command syntax * trying to fix azure pipeline image and raja plugin flag * make edits to azure pipeline file * Switching from quartz to ruby in gitlab CI * Updating umpire camp dependency * Release v2022.03.0 (#208) * Update Spack packages for CI (#210) * Add CHAI + RAJA launch integration test. (#219) * Release v2022.10.0 (#217) * Make operator overloads bind in more cases (#222) * MigrateInner on CHAI Copyable for PINNED / UM (#223) * Use shared CI (#213) * Deprecate API for synchronization after device kernels (#228) * setExecutionSpace thread safety(#230) Co-authored-by: Neela Kausik * Release v2023.06.0 (#232) * [Woptim] Update shared ci to v2023.08.0 (#236) * Remove explicit device synchronization (#235) * Update radiuss-shared-ci to new release (with radiuss-spack-configs) (#238) * Use free instead of realloc to zero size (#239) * [Woptim] shared ci 2023.12.0 (#241) * Update Shared CI and Spack Configs to add poodle * Activate poodle subpipeline * Reduce allocated time * Fine tune allocation duration * Fine tune allocation duration in CI * Comment alloc command choices * Update RSC to main * Fix missing poodle alloc info, update to radiuss Shared CI 2023.12.0, sync build_and_test script with RAJA * Update radiuss-spack-configs * CI updates (#244) * Updates radiuss-spack-configs to commit a37c3e1 * Changes to match the CI of other projects * Support for a single memory space writeable by both GPU and CPU (#242) * Support for a single memory space writeable by both GPU and CPU. Provides appropriate synchronizations when needed instead of performing any memory copies. --------- Co-authored-by: Alan Dayton * Clean up CHAI dependency handling * Revert "Clean up CHAI dependency handling" This reverts commit 5afb66ab6fa12d2d09e846c676696d420f4f7f69. * Clean up CMake in CHAI (#247) * Update copyright to 2024 (#248) * Update to BLT v0.6.1 (#243) * Update to BLT v0.6.1 * Update to RAJA v2024.02.0 * Update to Umpire 2024.02.0 * Use new way of exporting tpls * Disable failing container builds since container images are being revamped * Allow failure for the job using libcpp - doesn't play nicely with fmt at the moment * Use branch of radiuss-spack-configs to fix spack builds * Add install tests * Add a file with release notes * Ignoring runner issues on tioga since the jobs passed previously * [Woptim] update spack + update to rocm 5.7.1 (#245) * Update Spack version * Update RADIUSS Spack Configs with February 2024 release changes --------- Co-authored-by: Alan Dayton * Update version numbers --------- Co-authored-by: David Beckingsale Co-authored-by: Benjamin T. Liu Co-authored-by: Kristi Co-authored-by: Kristi Co-authored-by: Arturo Vargas Co-authored-by: robinson96 Co-authored-by: Adrien Bernede <51493078+adrienbernede@users.noreply.github.com> Co-authored-by: Neela Kausik Co-authored-by: Ben Liu <38140930+liu15@users.noreply.github.com> --- .gitignore | 1 + .gitlab-ci.yml | 126 +++--- .gitlab/butte-jobs.yml | 71 --- .gitlab/butte-templates.yml | 31 -- .gitlab/custom-jobs-and-variables.yml | 73 ++++ .gitlab/jobs/corona.yml | 28 ++ .gitlab/jobs/lassen.yml | 62 +++ .gitlab/jobs/poodle.yml | 33 ++ .gitlab/jobs/ruby.yml | 33 ++ .gitlab/jobs/tioga.yml | 28 ++ .gitlab/lassen-jobs.yml | 80 ---- .gitlab/lassen-templates.yml | 30 -- .gitlab/quartz-jobs.yml | 37 -- .gitlab/quartz-templates.yml | 54 --- .gitlab/subscribed-pipelines.yml | 111 +++++ .uberenv_config.json | 13 +- CMakeLists.txt | 91 ++-- Dockerfile | 143 +++--- LICENSE | 2 +- README.md | 10 +- RELEASE_NOTES.md | 27 ++ azure-pipelines.yml | 89 ++-- benchmarks/CMakeLists.txt | 10 +- benchmarks/chai_arraymanager_benchmarks.cpp | 4 +- benchmarks/chai_benchmark_utils.hpp | 4 +- benchmarks/chai_managed_ptr_benchmarks.cpp | 4 +- benchmarks/chai_managedarray_benchmarks.cpp | 4 +- blt | 2 +- cmake/ChaiBasics.cmake | 9 +- cmake/SetupChaiOptions.cmake | 46 ++ cmake/chai-config.cmake.in | 33 ++ cmake/thirdparty/SetupChaiThirdparty.cmake | 87 ++-- docs/CMakeLists.txt | 4 +- docs/doxygen/CMakeLists.txt | 4 +- docs/doxygen/Doxyfile.in | 16 +- docs/sphinx/CMakeLists.txt | 46 +- docs/sphinx/advanced_configuration.rst | 22 +- docs/sphinx/code_documentation.rst | 6 + docs/sphinx/conf.py | 8 +- docs/sphinx/conf.py.in | 8 +- docs/sphinx/contribution_guide.rst | 6 + docs/sphinx/developer/ci.rst | 6 + docs/sphinx/developer/uberenv.rst | 18 +- docs/sphinx/developer_guide.rst | 6 + docs/sphinx/getting_started.rst | 6 + docs/sphinx/index.rst | 45 +- docs/sphinx/tutorial.rst | 6 + docs/sphinx/user_guide.rst | 45 +- examples/CMakeLists.txt | 19 +- examples/chai-umpire-allocators.cpp | 4 +- examples/ex1.cpp | 4 +- examples/example.cpp | 46 +- examples/managed_ptr_example.cpp | 52 +++ examples/pinned.cpp | 4 +- reproducers/CMakeLists.txt | 33 ++ reproducers/managed_ptr_reproducer.cpp | 54 +++ .../virtual_function_complex_reproducer.cpp | 84 ++++ .../virtual_function_simple_reproducer.cpp | 41 ++ scripts/apply-license-info.sh | 4 +- scripts/check-license-info.sh | 33 ++ scripts/format-source.sh | 4 +- scripts/gitlab/build_and_test.sh | 188 ++++++-- scripts/license.txt | 8 +- scripts/make_release_tarball.sh | 6 +- scripts/radiuss-spack-configs | 2 +- scripts/spack_packages/chai/package.py | 281 ------------ scripts/spack_packages/raja/package.py | 336 -------------- .../umpire/camp_target_umpire_3.0.0.patch | 18 - scripts/spack_packages/umpire/package.py | 396 ----------------- scripts/travis/build_and_test.sh | 4 +- scripts/travis/install_llvm.sh | 4 +- scripts/uberenv | 2 +- scripts/update-copyright-year.sh | 39 -- scripts/update-copyright.sh | 68 +++ src/CMakeLists.txt | 4 +- src/chai/ArrayManager.cpp | 105 +++-- src/chai/ArrayManager.hpp | 28 +- src/chai/ArrayManager.inl | 6 +- src/chai/CMakeLists.txt | 50 ++- src/chai/ChaiMacros.hpp | 4 +- src/chai/ExecutionSpaces.hpp | 4 +- src/chai/ManagedArray.hpp | 31 +- src/chai/ManagedArray.inl | 90 ++-- src/chai/ManagedArrayView.hpp | 4 +- src/chai/ManagedArray_thin.inl | 139 ++++-- src/chai/PointerRecord.hpp | 4 +- src/chai/RajaExecutionSpacePlugin.cpp | 4 +- src/chai/RajaExecutionSpacePlugin.hpp | 4 +- src/chai/Types.hpp | 4 +- src/chai/chai-config.cmake.in | 13 - src/chai/config.hpp.in | 6 +- src/chai/managed_array.hpp | 4 +- src/chai/managed_ptr.hpp | 144 +++--- src/chai/pluginLinker.hpp | 4 +- src/tpl/raja | 2 +- src/tpl/umpire | 2 +- src/util/forall.hpp | 23 +- tests/CMakeLists.txt | 7 +- tests/install/CMakeLists.txt | 16 + tests/install/using-with-cmake/CMakeLists.txt | 51 +++ .../using-with-cmake/host-config.cmake.in | 54 +++ .../using-with-cmake/using-with-cmake.cpp | 21 + tests/integration/CMakeLists.txt | 37 +- tests/integration/managed_array_tests.cpp | 4 +- tests/integration/managed_ptr_tests.cpp | 4 +- tests/integration/raja-chai-launch.cpp | 409 ++++++++++++++++++ tests/integration/raja-chai-nested.cpp | 4 +- tests/integration/raja-chai-tests.cpp | 4 +- tests/unit/CMakeLists.txt | 8 +- tests/unit/array_manager_unit_tests.cpp | 4 +- tests/unit/managed_array_unit_tests.cpp | 4 +- tests/unit/managed_ptr_unit_tests.cpp | 4 +- 112 files changed, 2419 insertions(+), 2253 deletions(-) delete mode 100644 .gitlab/butte-jobs.yml delete mode 100644 .gitlab/butte-templates.yml create mode 100644 .gitlab/custom-jobs-and-variables.yml create mode 100644 .gitlab/jobs/corona.yml create mode 100644 .gitlab/jobs/lassen.yml create mode 100644 .gitlab/jobs/poodle.yml create mode 100644 .gitlab/jobs/ruby.yml create mode 100644 .gitlab/jobs/tioga.yml delete mode 100644 .gitlab/lassen-jobs.yml delete mode 100644 .gitlab/lassen-templates.yml delete mode 100644 .gitlab/quartz-jobs.yml delete mode 100644 .gitlab/quartz-templates.yml create mode 100644 .gitlab/subscribed-pipelines.yml create mode 100644 RELEASE_NOTES.md create mode 100644 cmake/SetupChaiOptions.cmake create mode 100644 cmake/chai-config.cmake.in create mode 100644 examples/managed_ptr_example.cpp create mode 100644 reproducers/CMakeLists.txt create mode 100644 reproducers/managed_ptr_reproducer.cpp create mode 100644 reproducers/virtual_function_complex_reproducer.cpp create mode 100644 reproducers/virtual_function_simple_reproducer.cpp create mode 100755 scripts/check-license-info.sh delete mode 100644 scripts/spack_packages/chai/package.py delete mode 100644 scripts/spack_packages/raja/package.py delete mode 100644 scripts/spack_packages/umpire/camp_target_umpire_3.0.0.patch delete mode 100644 scripts/spack_packages/umpire/package.py delete mode 100755 scripts/update-copyright-year.sh create mode 100755 scripts/update-copyright.sh delete mode 100644 src/chai/chai-config.cmake.in create mode 100644 tests/install/CMakeLists.txt create mode 100644 tests/install/using-with-cmake/CMakeLists.txt create mode 100644 tests/install/using-with-cmake/host-config.cmake.in create mode 100644 tests/install/using-with-cmake/using-with-cmake.cpp create mode 100644 tests/integration/raja-chai-launch.cpp diff --git a/.gitignore b/.gitignore index 7987e0f1..e9572eae 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,4 @@ Win32/chai.vcxproj.user Win32/.vs/ Win32/x64/ *.aps +*.orig diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 5a9da66e..9908db90 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,80 +1,90 @@ ############################################################################### -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: (MIT) ############################################################################### +# DESCRIPTION: ############################################################################### # General GitLab pipelines configurations for supercomputers and Linux clusters # at Lawrence Livermore National Laboratory (LLNL). -# # This entire pipeline is LLNL-specific -# ############################################################################# - -# We define the following GitLab pipeline variables: # -# GIT_SUBMODULE_STRATEGY: -# Tells Gitlab to recursively update the submodules when cloning the project +# Important note: This file is a template provided by llnl/radiuss-shared-ci. +# Remains to set variable values, change the reference to the radiuss-shared-ci +# repo, opt-in and out optional features. The project can then extend it with +# additional stages. # -# ALLOC_NAME: -# On LLNL's quartz, this pipeline creates only one allocation shared among jobs -# in order to save time and resources. This allocation has to be uniquely named -# so that we are sure to retrieve it. +# In addition, each project should copy over and complete: +# - .gitlab/custom-jobs-and-variables.yml +# - .gitlab/subscribed-pipelines.yml # -# BUILD_ROOT: -# The path to the shared resources between all jobs. The BUILD_ROOT is unique to -# the pipeline, preventing any form of concurrency with other pipelines. This -# also means that the BUILD_ROOT directory will never be cleaned. -# TODO: add a clean-up mechanism +# The jobs should be specified in a file local to the project, +# - .gitlab/jobs/${CI_MACHINE}.yml +# or generated (see LLNL/Umpire for an example). +############################################################################### +# We define the following GitLab pipeline variables: variables: +##### LC GITLAB CONFIGURATION +# Use an LLNL service user to run CI. This prevents from running pipelines as an +# actual user. + LLNL_SERVICE_USER: umdev +# Use the service user workspace. Solves permission issues, stores everything +# at the same location whoever triggers a pipeline. + CUSTOM_CI_BUILDS_DIR: /usr/workspace/umdev/gitlab-runner +# Tells Gitlab to recursively update the submodules when cloning the project. GIT_SUBMODULE_STRATEGY: recursive - ALLOC_NAME: ${CI_PROJECT_NAME}_ci_${CI_PIPELINE_ID} - BUILD_ROOT: ${CI_PROJECT_DIR} - -# Normally, stages are blocking in Gitlab. However, using the keyword "needs" we -# can express dependencies between job that break the ordering of stages, in -# favor of a DAG. -# In practice q_*, l_* and b_* stages are independently run and start immediately. -stages: - - q_allocate_resources - - q_build_and_test - - q_release_resources - - l_build_and_test - - b_build_and_test +##### PROJECT VARIABLES +# We build the projects in the CI clone directory (used in +# script/gitlab/build_and_test.sh script). +# TODO: add a clean-up mechanism. + BUILD_ROOT: ${CI_PROJECT_DIR} -# This is the rules that drives the activation of "advanced" jobs. All advanced -# jobs will share this through a template mechanism. -.advanced_pipeline: - rules: - - if: '$CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop" || $ALL_TARGETS == "ON"' #run only if ... +##### SHARED_CI CONFIGURATION +# Required information about GitHub repository + GITHUB_PROJECT_NAME: "CHAI" + GITHUB_PROJECT_ORG: "LLNL" +# Set the build-and-test command. +# Nested variables are allowed and useful to customize the job command. We +# prevent variable expansion so that you can define them at job level. + JOB_CMD: + value: "./scripts/gitlab/build_and_test.sh" + expand: false +# Override the pattern describing branches that will skip the "draft PR filter +# test". Add protected branches here. See default value in +# preliminary-ignore-draft-pr.yml. +# ALWAYS_RUN_PATTERN: "" -# These are also templates (.name) that define project specific build commands. -# If an allocation exist with the name defined in this pipeline, the job will -# use it (slurm specific). -.build_toss_3_x86_64_ib_script: - script: - - echo ${ALLOC_NAME} - - export JOBID=$(squeue -h --name=${ALLOC_NAME} --format=%A) - - echo ${JOBID} - - srun $( [[ -n "${JOBID}" ]] && echo "--jobid=${JOBID}" ) -t 10 -N 1 scripts/gitlab/build_and_test.sh +# We organize the build-and-test stage with sub-pipelines. Each sub-pipeline +# corresponds to a test batch on a given machine. -# Lassen and Butte use a different job scheduler (spectrum lsf) that does not -# allow pre-allocation the same way slurm does. -.build_blueos_3_ppc64le_ib_script: - script: - - lalloc 1 -W 20 scripts/gitlab/build_and_test.sh +# High level stages +stages: + - prerequisites + - build-and-test -.build_blueos_3_ppc64le_ib_p9_script: - extends: .build_blueos_3_ppc64le_ib_script +# Template for jobs triggering a build-and-test sub-pipeline: +.build-and-test: + stage: build-and-test + trigger: + include: + - local: '.gitlab/custom-jobs-and-variables.yml' + - project: 'radiuss/radiuss-shared-ci' + ref: 'v2023.12.0' + file: 'pipelines/${CI_MACHINE}.yml' + - artifact: '${CI_MACHINE}-jobs.yml' + job: 'generate-job-lists' + strategy: depend + forward: + pipeline_variables: true -# This is where jobs are included. include: - - local: .gitlab/quartz-templates.yml - - local: .gitlab/quartz-jobs.yml - - local: .gitlab/lassen-templates.yml - - local: .gitlab/lassen-jobs.yml - - local: .gitlab/butte-templates.yml - - local: .gitlab/butte-jobs.yml + # [Optional] checks preliminary to running the actual CI test + #- project: 'radiuss/radiuss-shared-ci' + # ref: 'v2023.12.0' + # file: 'preliminary-ignore-draft-pr.yml' + # pipelines subscribed by the project + - local: '.gitlab/subscribed-pipelines.yml' diff --git a/.gitlab/butte-jobs.yml b/.gitlab/butte-jobs.yml deleted file mode 100644 index 45f3fe00..00000000 --- a/.gitlab/butte-jobs.yml +++ /dev/null @@ -1,71 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## - -#### -# All butte build jobs - -clang_default (build and test on butte): - variables: - SPEC: "%clang@default" - extends: .build_and_test_on_butte_advanced - -gcc_default (build and test on butte): - variables: - SPEC: "%gcc@default" - extends: .build_and_test_on_butte_advanced - -xl_default (build and test on butte): - variables: - SPEC: "%xl@default" - extends: .build_and_test_on_butte_advanced - -pgi_default (build and test on butte): - variables: - SPEC: "%pgi@default" - extends: .build_and_test_on_butte_advanced - allow_failure: true - -gcc_8_3_1 (build and test on butte): - variables: - SPEC: "%gcc@8.3.1" - extends: .build_and_test_on_butte_advanced - -clang_3_9_1 (build and test on butte): - variables: - SPEC: "%clang@3.9.1" - extends: .build_and_test_on_butte_advanced - -clang_4_0_0 (build and test on butte): - variables: - SPEC: "%clang@4.0.0" - extends: .build_and_test_on_butte_advanced - -clang_9_0_0 (build and test on butte): - variables: - SPEC: "%clang@9.0.0" - extends: .build_and_test_on_butte_advanced - -clang_coral_2018_08_08 (build and test on butte): - variables: - SPEC: "%clang@coral2018.08.08" - extends: .build_and_test_on_butte_advanced - allow_failure: true - -nvcc_gcc_4_9_3 (build and test on butte): - variables: - SPEC: "%gcc@4.9.3+cuda" - extends: .build_and_test_on_butte_advanced - -nvcc_clang_coral_2018_08_08 (build and test on butte): - variables: - SPEC: "%clang@coral2018.08.08+cuda" - extends: .build_and_test_on_butte_advanced - -nvcc_xl-beta-2019.06.20 (build and test on butte): - variables: - SPEC: "%xl@beta2019.06.20+cuda" - extends: .build_and_test_on_butte_advanced diff --git a/.gitlab/butte-templates.yml b/.gitlab/butte-templates.yml deleted file mode 100644 index 462b9cde..00000000 --- a/.gitlab/butte-templates.yml +++ /dev/null @@ -1,31 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## - -#### -# Shared configuration of jobs for butte -.on_butte_advanced: - variables: - tags: - - shell - - butte - rules: - - if: '$ON_BUTTE != "ON"' #run except if ... - when: never - - when: on_success - allow_failure: true - -.build_and_test_on_butte: - stage: b_build_and_test - extends: [.build_blueos_3_ppc64le_ib_script, .on_butte_advanced] - needs: [] - -# Note: .build_and_test_on_butte_advanced inherits from -# .build_and_test_on_butte_advanced and .advanced_pileline. -# In particular, the rules section will be merged. Careful when changing rules. -.build_and_test_on_butte_advanced: - extends: [.build_and_test_on_butte, .advanced_pipeline] - allow_failure: true diff --git a/.gitlab/custom-jobs-and-variables.yml b/.gitlab/custom-jobs-and-variables.yml new file mode 100644 index 00000000..c4c20dac --- /dev/null +++ b/.gitlab/custom-jobs-and-variables.yml @@ -0,0 +1,73 @@ +############################################################################### +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################### + +# We define the following GitLab pipeline variables: +variables: +# On LLNL's ruby, this pipeline creates only one allocation shared among jobs +# in order to save time and resources. This allocation has to be uniquely named +# so that we are sure to retrieve it and avoid collisions. + ALLOC_NAME: ${CI_PROJECT_NAME}_ci_${CI_PIPELINE_ID} + +# Ruby +# Arguments for top level allocation + RUBY_SHARED_ALLOC: "--exclusive --reservation=ci --time=7 --nodes=1" +# Arguments for job level allocation +# Note: We repeat the reservation, necessary when jobs are manually re-triggered. + RUBY_JOB_ALLOC: "--overlap --reservation=ci --nodes=1" +# Project specific variants for ruby + PROJECT_RUBY_VARIANTS: "~shared +raja tests=basic " +# Project specific deps for ruby + PROJECT_RUBY_DEPS: "" + +# Poodle +# Arguments for top level allocation + POODLE_SHARED_ALLOC: "--exclusive --partition=pdebug --time=7 --nodes=1" +# Arguments for job level allocation +# Note: We repeat the reservation, necessary when jobs are manually re-triggered. + POODLE_JOB_ALLOC: "--overlap --nodes=1" +# Project specific variants for poodle + PROJECT_POODLE_VARIANTS: "~shared +raja tests=basic " +# Project specific deps for poodle + PROJECT_POODLE_DEPS: "" + +# Corona +# Arguments for top level allocation + CORONA_SHARED_ALLOC: "--exclusive --time-limit=9m --nodes=1" +# Arguments for job level allocation + CORONA_JOB_ALLOC: "--nodes=1 --begin-time=+5s" +# Project specific variants for corona + PROJECT_CORONA_VARIANTS: "~shared +raja tests=basic " +# Project specific deps for corona + PROJECT_CORONA_DEPS: "" + +# Tioga +# Arguments for top level allocation + TIOGA_SHARED_ALLOC: "--exclusive --time-limit=16m --nodes=1" +# Arguments for job level allocation + TIOGA_JOB_ALLOC: "--nodes=1 --begin-time=+5s" +# Project specific variants for tioga + PROJECT_TIOGA_VARIANTS: "~shared +raja tests=basic " +# Project specific deps for tioga + PROJECT_TIOGA_DEPS: "" + +# Lassen and Butte use a different job scheduler (spectrum lsf) that does not +# allow pre-allocation the same way slurm does. +# Arguments for job level allocation + LASSEN_JOB_ALLOC: "1 -W 8 -q pci" +# Project specific variants for lassen + PROJECT_LASSEN_VARIANTS: "~shared +raja cuda_arch=70 tests=basic" +# Project specific deps for lassen + PROJECT_LASSEN_DEPS: "" + +# Configuration shared by build and test jobs specific to this project. +# Not all configuration can be shared. Here projects can fine tune the +# CI behavior. +# See Umpire for an example (export junit test reports). +.custom_job: + artifacts: + reports: + junit: junit.xml diff --git a/.gitlab/jobs/corona.yml b/.gitlab/jobs/corona.yml new file mode 100644 index 00000000..390ce1fc --- /dev/null +++ b/.gitlab/jobs/corona.yml @@ -0,0 +1,28 @@ +############################################################################## +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## + +# Override reproducer section to define project specific variables. +.corona_reproducer_vars: &corona_reproducer_vars + - | + echo -e "export MODULE_LIST=\"${MODULE_LIST}\"" + echo -e "export SPEC=\"${SPEC//\"/\\\"}\"" + +######################## +# Overridden shared jobs +######################## +# We duplicate the shared jobs description and add necessary changes for this +# project. We keep ${PROJECT__VARIANTS} and ${PROJECT__DEPS} +# So that the comparison with the original job is easier. + +# No overridden jobs so far. + +############ +# Extra jobs +############ +# We do not recommend using ${PROJECT__VARIANTS} and +# ${PROJECT__DEPS} in the extra jobs. There is no reason not to fully +# describe the spec here. diff --git a/.gitlab/jobs/lassen.yml b/.gitlab/jobs/lassen.yml new file mode 100644 index 00000000..d8003b1d --- /dev/null +++ b/.gitlab/jobs/lassen.yml @@ -0,0 +1,62 @@ +############################################################################## +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## + +# Override reproducer section to define project specific variables. +.lassen_reproducer_vars: &lassen_reproducer_vars + - | + echo -e "export MODULE_LIST=\"${MODULE_LIST}\"" + echo -e "export SPEC=\"${SPEC//\"/\\\"}\"" + +######################## +# Overridden shared jobs +######################## +# We duplicate the shared jobs description and add necessary changes for this +# project. We keep ${PROJECT__VARIANTS} and ${PROJECT__DEPS} +# So that the comparison with the original job is easier. + +# Overridden to increase allocation time +xl_2022_08_19_gcc_8_3_1_cuda_11_2_0: + variables: + SPEC: "${PROJECT_LASSEN_VARIANTS} +cuda %xl@16.1.1.12.gcc.8.3.1 ^cuda@11.2.0+allow-unsupported-compilers ${PROJECT_LASSEN_DEPS}" + MODULE_LIST: "cuda/11.2.0" + LASSEN_JOB_ALLOC: "1 -W 12 -q pci" + extends: .job_on_lassen + + +# Override job with old cuda to allow it to fail. +clang_12_0_1_ibm_gcc_8_3_1_cuda_10_1_243: + variables: + SPEC: "${PROJECT_LASSEN_VARIANTS} +cuda %clang@12.0.1.ibm.gcc.8.3.1 ^cuda@10.1.243+allow-unsupported-compilers ${PROJECT_LASSEN_DEPS}" + extends: .job_on_lassen + allow_failure: true + + +############ +# Extra jobs +############ +# We do not recommend using ${PROJECT__VARIANTS} and +# ${PROJECT__DEPS} in the extra jobs. There is no reason not to fully +# describe the spec here. + +clang_12_0_1_libcpp: + extends: .job_on_lassen + variables: + SPEC: "~shared +raja tests=basic %clang@12.0.1 cflags==\"-DGTEST_HAS_CXXABI_H_=0\" cxxflags==\"-stdlib=libc++ -DGTEST_HAS_CXXABI_H_=0\"" + allow_failure: true + +clang_12_0_1_gcc_8_3_1_memleak: + extends: .job_on_lassen + variables: + SPEC: "~shared +raja tests=basic %clang@12.0.1.gcc.8.3.1 cxxflags==\"-fsanitize=address\"" + ASAN_OPTIONS: "detect_leaks=1" + +clang_12_0_1_datarace: + extends: .job_on_lassen + variables: + SPEC: "~shared +raja tests=basic %clang@12.0.1 cxxflags==\"-fsanitize=thread\"" + allow_failure: true + diff --git a/.gitlab/jobs/poodle.yml b/.gitlab/jobs/poodle.yml new file mode 100644 index 00000000..848cf125 --- /dev/null +++ b/.gitlab/jobs/poodle.yml @@ -0,0 +1,33 @@ +############################################################################### +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################### + +# Override reproducer section to define projet specific variables. +.poodle_reproducer_vars: &poodle_reproducer_vars + - | + echo -e "export MODULE_LIST=\"${MODULE_LIST}\"" + echo -e "export SPEC=\"${SPEC//\"/\\\"}\"" + +######################## +# Overridden shared jobs +######################## +# We duplicate the shared jobs description and add necessary changes for this +# project. We keep ${PROJECT__VARIANTS} and ${PROJECT__DEPS} +# So that the comparison with the original job is easier. + +# No overridden jobs so far. + +############ +# Extra jobs +############ +# We do not recommend using ${PROJECT__VARIANTS} and +# ${PROJECT__DEPS} in the extra jobs. There is no reason not to fully +# describe the spec here. + +clang_14_0_6_develop_tpls: + variables: + SPEC: " %clang@14.0.6 ^umpire@develop ^raja@develop ^camp@main" + extends: .job_on_poodle diff --git a/.gitlab/jobs/ruby.yml b/.gitlab/jobs/ruby.yml new file mode 100644 index 00000000..8e26fa3c --- /dev/null +++ b/.gitlab/jobs/ruby.yml @@ -0,0 +1,33 @@ +############################################################################### +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################### + +# Override reproducer section to define project specific variables. +.ruby_reproducer_vars: &ruby_reproducer_vars + - | + echo -e "export MODULE_LIST=\"${MODULE_LIST}\"" + echo -e "export SPEC=\"${SPEC//\"/\\\"}\"" + +######################## +# Overridden shared jobs +######################## +# We duplicate the shared jobs description and add necessary changes for this +# project. We keep ${PROJECT__VARIANTS} and ${PROJECT__DEPS} +# So that the comparison with the original job is easier. + +# No overridden jobs so far. + +############ +# Extra jobs +############ +# We do not recommend using ${PROJECT__VARIANTS} and +# ${PROJECT__DEPS} in the extra jobs. There is no reason not to fully +# describe the spec here. + +clang_14_0_6_develop_tpls: + variables: + SPEC: " %clang@14.0.6 ^umpire@develop ^raja@develop ^camp@main" + extends: .job_on_ruby diff --git a/.gitlab/jobs/tioga.yml b/.gitlab/jobs/tioga.yml new file mode 100644 index 00000000..27eb3c99 --- /dev/null +++ b/.gitlab/jobs/tioga.yml @@ -0,0 +1,28 @@ +############################################################################## +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## + +# Override reproducer section to define project specific variables. +.tioga_reproducer_vars: &tioga_reproducer_vars + - | + echo -e "export MODULE_LIST=\"${MODULE_LIST}\"" + echo -e "export SPEC=\"${SPEC//\"/\\\"}\"" + +######################## +# Overridden shared jobs +######################## +# We duplicate the shared jobs description and add necessary changes for this +# project. We keep ${PROJECT__VARIANTS} and ${PROJECT__DEPS} +# So that the comparison with the original job is easier. + +# No overridden jobs so far. + +############ +# Extra jobs +############ +# We do not recommend using ${PROJECT__VARIANTS} and +# ${PROJECT__DEPS} in the extra jobs. There is no reason not to fully +# describe the spec here. diff --git a/.gitlab/lassen-jobs.yml b/.gitlab/lassen-jobs.yml deleted file mode 100644 index 096a326c..00000000 --- a/.gitlab/lassen-jobs.yml +++ /dev/null @@ -1,80 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## - -# CPU ONLY -ibm_clang_9: - variables: - SPEC: "%clang@9.0.0ibm" - extends: .build_and_test_on_lassen - -ibm_clang_9_gcc_8: - variables: - SPEC: "%clang@9.0.0ibm cxxflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" - extends: .build_and_test_on_lassen - -gcc_8_3_1: - variables: - SPEC: "%gcc@8.3.1 ^cmake@3.14.5" - extends: .build_and_test_on_lassen - -xl_16_1_1_7: - variables: - SPEC: "%xl@16.1.1.7" - extends: .build_and_test_on_lassen - -xl_16_1_1_7_gcc_8_3_1: - variables: - SPEC: "%xl@16.1.1.7 cxxflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1" - extends: .build_and_test_on_lassen - -# CUDA - -ibm_clang_9_cuda: - variables: - SPEC: "+cuda %clang@9.0.0ibm ^cuda@10.1.168" - extends: .build_and_test_on_lassen - -ibm_clang_9_gcc_8_cuda: - variables: - SPEC: "+cuda %clang@9.0.0ibm cxxflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 ^cuda@10.1.168" - allow_failure: true - extends: .build_and_test_on_lassen - -gcc_8_3_1_cuda: - variables: - SPEC: "+cuda %gcc@8.3.1 ^cuda@10.1.168" - extends: .build_and_test_on_lassen - -xl_16_1_1_7_cuda: - variables: - SPEC: "+cuda %xl@16.1.1.7 ^cuda@10.1.168 ^cmake@3.14.5" - extends: .build_and_test_on_lassen - -xl_16_1_1_7_gcc_8_3_1_cuda_11: - variables: - SPEC: "+cuda %xl@16.1.1.7 cxxflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags=--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 ^cuda@11.0.2 ^cmake@3.14.5" - allow_failure: true - extends: .build_and_test_on_lassen - -# EXTRAS - -clang_9_0_0_libcpp (build and test on lassen): - variables: - SPEC: "%clang@9.0.0+libcpp" - extends: .build_and_test_on_lassen - -clang_9_0_0_memleak (build and test on lassen): - variables: - SPEC: "%clang@9.0.0 cxxflags=-fsanitize=address" - ASAN_OPTIONS: "detect_leaks=1" - extends: .build_and_test_on_lassen - -clang_9_0_0_datarace (build and test on lassen): - variables: - SPEC: "%clang@9.0.0 cxxflags=-fsanitize=thread" - allow_failure: true - extends: .build_and_test_on_lassen \ No newline at end of file diff --git a/.gitlab/lassen-templates.yml b/.gitlab/lassen-templates.yml deleted file mode 100644 index 16a8e992..00000000 --- a/.gitlab/lassen-templates.yml +++ /dev/null @@ -1,30 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## - -#### -# Shared configuration of jobs for lassen -.on_lassen: - variables: - tags: - - shell - - lassen - rules: - - if: '$CI_COMMIT_BRANCH =~ /_lnone/ || $ON_LASSEN == "OFF"' #run except if ... - when: never - - when: on_success - -.build_and_test_on_lassen: - stage: l_build_and_test - extends: [.build_blueos_3_ppc64le_ib_p9_script, .on_lassen] - needs: [] - -# Note: .build_and_test_on_lassen_advanced inherits from -# .build_and_test_on_lassen and .advanced_pileline. -# In particular, the rules section will be merged. Careful when changing rules. -.build_and_test_on_lassen_advanced: - extends: [.build_and_test_on_lassen, .advanced_pipeline] - diff --git a/.gitlab/quartz-jobs.yml b/.gitlab/quartz-jobs.yml deleted file mode 100644 index 0c2cc055..00000000 --- a/.gitlab/quartz-jobs.yml +++ /dev/null @@ -1,37 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## -clang_10: - variables: - SPEC: " %clang@10.0.1" - extends: .build_and_test_on_quartz - -clang_9: - variables: - SPEC: " %clang@9.0.0" - extends: .build_and_test_on_quartz - -gcc_8_1_0: - variables: - SPEC: "%gcc@8.1.0" - extends: .build_and_test_on_quartz - -# icpc_18_0_2: -# variables: -# SPEC: " tests=none %intel@18.0.2" -# extends: .build_and_test_on_quartz - -icpc_19_1_0: - variables: - SPEC: "%intel@19.1.0" - extends: .build_and_test_on_quartz - -# EXTRAS - -gcc_4_9_3: - variables: - SPEC: "%gcc@4.9.3" - extends: .build_and_test_on_quartz diff --git a/.gitlab/quartz-templates.yml b/.gitlab/quartz-templates.yml deleted file mode 100644 index 459bf121..00000000 --- a/.gitlab/quartz-templates.yml +++ /dev/null @@ -1,54 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and Umpire -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: (MIT) -############################################################################## - -#### -# This is the share configuration of jobs for quartz - -#### -# In pre-build phase, allocate a node for builds -.on_quartz: - tags: - - shell - - quartz - rules: - - if: '$CI_COMMIT_BRANCH =~ /_qnone/ || $ON_QUARTZ == "OFF"' #run except if ... - when: never - - if: '$CI_JOB_NAME =~ /release_resources/' - when: always - - when: on_success - -#### -# In pre-build phase, allocate a node for builds -allocate_resources (on quartz): - variables: - GIT_STRATEGY: none - extends: .on_quartz - stage: q_allocate_resources - script: - - salloc -N 1 -c 36 -p pdebug -t 20 --no-shell --job-name=${ALLOC_NAME} - -#### -# In post-build phase, deallocate resources -# Note : make sure this is run even on build phase failure -release_resources (on quartz): - variables: - GIT_STRATEGY: none - extends: .on_quartz - stage: q_release_resources - script: - - export JOBID=$(squeue -h --name=${ALLOC_NAME} --format=%A) - - ([[ -n "${JOBID}" ]] && scancel ${JOBID}) - -#### -# Generic quartz build job, extending build script -.build_and_test_on_quartz: - extends: [.build_toss_3_x86_64_ib_script, .on_quartz] - stage: q_build_and_test - -.build_and_test_on_quartz_advanced: - extends: [.build_and_test_on_quartz, .advanced_pipeline] - diff --git a/.gitlab/subscribed-pipelines.yml b/.gitlab/subscribed-pipelines.yml new file mode 100644 index 00000000..dd317a75 --- /dev/null +++ b/.gitlab/subscribed-pipelines.yml @@ -0,0 +1,111 @@ +############################################################################### +# Copyright (c) 2022-24, Lawrence Livermore National Security, LLC and RADIUSS +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################### + +# The template job to test whether a machine is up. +# Expects CI_MACHINE defined to machine name. +.machine-check: + stage: prerequisites + tags: [shell, oslic] + variables: + GIT_STRATEGY: none + script: + - | + if [[ $(jq '.[env.CI_MACHINE].total_nodes_up' /usr/global/tools/lorenz/data/loginnodeStatus) == 0 ]] + then + echo -e "\e[31mNo node available on ${CI_MACHINE}\e[0m" + curl --url "https://api.github.com/repos/${GITHUB_PROJECT_ORG}/${GITHUB_PROJECT_NAME}/statuses/${CI_COMMIT_SHA}" \ + --header 'Content-Type: application/json' \ + --header "authorization: Bearer ${GITHUB_TOKEN}" \ + --data "{ \"state\": \"failure\", \"target_url\": \"${CI_PIPELINE_URL}\", \"description\": \"GitLab ${CI_MACHINE} down\", \"context\": \"ci/gitlab/${CI_MACHINE}\" }" + exit 1 + fi + +### +# Trigger a build-and-test pipeline for a machine. +# Comment the jobs for machines you don’t need. +### + +# One job to generate the job list for all the subpipelines +generate-job-lists: + stage: prerequisites + tags: [shell, oslic] + variables: + RADIUSS_JOBS_PATH: "scripts/radiuss-spack-configs/gitlab/radiuss-jobs" + LOCAL_JOBS_PATH: ".gitlab/jobs" + script: + - cat ${RADIUSS_JOBS_PATH}/ruby.yml ${LOCAL_JOBS_PATH}/ruby.yml > ruby-jobs.yml + - cat ${RADIUSS_JOBS_PATH}/poodle.yml ${LOCAL_JOBS_PATH}/poodle.yml > poodle-jobs.yml + - cat ${RADIUSS_JOBS_PATH}/lassen.yml ${LOCAL_JOBS_PATH}/lassen.yml > lassen-jobs.yml + - cat ${RADIUSS_JOBS_PATH}/corona.yml ${LOCAL_JOBS_PATH}/corona.yml > corona-jobs.yml + - cat ${RADIUSS_JOBS_PATH}/tioga.yml ${LOCAL_JOBS_PATH}/tioga.yml > tioga-jobs.yml + artifacts: + paths: + - ruby-jobs.yml + - poodle-jobs.yml + - lassen-jobs.yml + - corona-jobs.yml + - tioga-jobs.yml + +# RUBY +ruby-up-check: + variables: + CI_MACHINE: "ruby" + extends: [.machine-check] + +ruby-build-and-test: + variables: + CI_MACHINE: "ruby" + needs: [ruby-up-check, generate-job-lists] + extends: [.build-and-test] + +# POODLE +poodle-up-check: + variables: + CI_MACHINE: "poodle" + extends: [.machine-check] + +poodle-build-and-test: + variables: + CI_MACHINE: "poodle" + needs: [poodle-up-check, generate-job-lists] + extends: [.build-and-test] + +# CORONA +corona-up-check: + variables: + CI_MACHINE: "corona" + extends: [.machine-check] + +corona-build-and-test: + variables: + CI_MACHINE: "corona" + needs: [corona-up-check, generate-job-lists] + extends: [.build-and-test] + +# TIOGA +tioga-up-check: + variables: + CI_MACHINE: "tioga" + extends: [.machine-check] + +tioga-build-and-test: + variables: + CI_MACHINE: "tioga" + needs: [tioga-up-check, generate-job-lists] + extends: [.build-and-test] + +# LASSEN +lassen-up-check: + variables: + CI_MACHINE: "lassen" + extends: [.machine-check] + +lassen-build-and-test: + variables: + CI_MACHINE: "lassen" + needs: [lassen-up-check, generate-job-lists] + extends: [.build-and-test] diff --git a/.uberenv_config.json b/.uberenv_config.json index e3553570..3792b3ad 100644 --- a/.uberenv_config.json +++ b/.uberenv_config.json @@ -1,11 +1,14 @@ { "package_name" : "chai", "package_version" : "develop", -"package_final_phase" : "hostconfig", +"package_final_phase" : "initconfig", "package_source_dir" : "../..", -"spack_url": "https://github.com/davidbeckingsale/spack", -"spack_branch": "feature/allow-untested-cuda-versions", -"spack_commit": "46b22d0f6227f6b12bab712bda5b916a53cfc67d", +"spack_url": "https://github.com/spack/spack.git", +"spack_branch": "develop-2024-01-21", "spack_activate" : {}, "spack_configs_path": "scripts/radiuss-spack-configs", -"spack_packages_path": "scripts/spack_packages"} +"spack_packages_path": "scripts/radiuss-spack-configs/packages", +"spack_concretizer": "clingo", +"spack_setup_clingo": false, +"debug": true +} diff --git a/CMakeLists.txt b/CMakeLists.txt index 9ac5b454..a21c5518 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## @@ -8,51 +8,22 @@ cmake_policy(SET CMP0057 NEW) cmake_policy(SET CMP0048 NEW) cmake_policy(SET CMP0025 NEW) -project(Chai LANGUAGES CXX VERSION 2.4.0) - -set(ENABLE_CUDA Off CACHE BOOL "Enable CUDA") -set(ENABLE_HIP Off CACHE BOOL "Enable HIP") -set(ENABLE_GPU_SIMULATION_MODE Off CACHE BOOL "Enable GPU Simulation Mode") -set(ENABLE_OPENMP OFF CACHE BOOL "Enable OpenMP") -set(ENABLE_MPI Off CACHE BOOL "Enable MPI (for umpire replay only)") -set(ENABLE_BENCHMARKS On CACHE BOOL "Enable benchmarks") -option(ENABLE_IMPLICIT_CONVERSIONS "Enable implicit conversions to-from raw pointers" On) -option(DISABLE_RM "Make ManagedArray a thin wrapper" Off) -mark_as_advanced(DISABLE_RM) -option(ENABLE_UM "Use CUDA unified (managed) memory" Off) -option(ENABLE_PINNED "Use pinned host memory" Off) -option(ENABLE_RAJA_PLUGIN "Build plugin to set RAJA execution spaces" Off) -option(CHAI_ENABLE_GPU_ERROR_CHECKING "Enable GPU error checking" On) -option(CHAI_ENABLE_MANAGED_PTR "Enable managed_ptr" On) -option(CHAI_DEBUG "Enable Debug Logging.") -set(ENABLE_RAJA_NESTED_TEST ON CACHE BOOL "Enable raja-chai-nested-tests, which fails to build on Debug CUDA builds.") - -set(ENABLE_TESTS On CACHE BOOL "") -set(ENABLE_EXAMPLES On CACHE BOOL "") -set(ENABLE_DOCS Off CACHE BOOL "") - -# options for Umpire as TPL -set(ENABLE_GMOCK On CACHE BOOL "") -set(ENABLE_ASSERTS "Build Umpire with assert() enabled" On) -set(ENABLE_GTEST_DEATH_TESTS ${ENABLE_ASSERTS} CACHE BOOL "") - -if (ENABLE_UM AND NOT ENABLE_CUDA) - message(FATAL_ERROR "Option ENABLE_UM requires ENABLE_CUDA") -endif() +include(CMakeDependentOption) +include(CMakePackageConfigHelpers) + +project(Chai LANGUAGES C CXX VERSION 2024.02.0) +cmake_minimum_required(VERSION 3.14) -set(ENABLE_COPY_HEADERS Off CACHE BOOL "") -set(BLT_CXX_STD c++11 CACHE STRING "") +include(cmake/SetupChaiOptions.cmake) + +set(BLT_CXX_STD c++14 CACHE STRING "") if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "PGI") message(STATUS "Setting CMAKE_CXX_EXTENSIONS to ON for PGI Compiler") set( CMAKE_CXX_EXTENSIONS ON ) endif() -if (ENABLE_CUDA) - cmake_minimum_required(VERSION 3.9) -else () - cmake_minimum_required(VERSION 3.8) -endif() +message(STATUS "Using CMake version ${CMAKE_VERSION}") ################################ # BLT @@ -77,22 +48,52 @@ if (NOT BLT_LOADED) include(${BLT_SOURCE_DIR}/SetupBLT.cmake) endif() +####################################### +# Options that depend on BLT Options +####################################### +cmake_dependent_option( CHAI_ENABLE_CUDA "Build CHAI with CUDA support" On + "ENABLE_CUDA" Off ) +cmake_dependent_option( CHAI_ENABLE_HIP "Build CHAI with HIP" On + "ENABLE_HIP" Off ) +cmake_dependent_option( CHAI_ENABLE_OPENMP "Build CHAI with OpenMP" On + "ENABLE_OPENMP" Off ) +cmake_dependent_option( CHAI_ENABLE_MPI "Build CHAI with MPI" On + "ENABLE_MPI" Off ) + +cmake_dependent_option(CHAI_ENABLE_TESTS "Build CHAI tests" On + "ENABLE_TESTS" Off) +cmake_dependent_option(CHAI_ENABLE_BENCHMARKS "Build CHAI benchmarks" On + "ENABLE_BENCHMARKS" Off) +cmake_dependent_option(CHAI_ENABLE_EXAMPLES "Build CHAI examples" On + "ENABLE_EXAMPLES" Off ) + +cmake_dependent_option(CHAI_ENABLE_DOCS "Build CHAI docs" On + "ENABLE_DOCS" Off) +cmake_dependent_option( CHAI_ENABLE_GMOCK "Build CHAI with gmock" On + "ENABLE_GMOCK" Off ) +cmake_dependent_option(CHAI_ENABLE_REPRODUCERS "Build CHAI reproducers" On + "ENABLE_REPRODUCERS" Off) + include(cmake/ChaiBasics.cmake) add_subdirectory(src) -if (ENABLE_TESTS) +if (CHAI_ENABLE_TESTS) add_subdirectory(tests) endif() -if (ENABLE_BENCHMARKS) +if (CHAI_ENABLE_BENCHMARKS) add_subdirectory(benchmarks) endif() -if (ENABLE_EXAMPLES) +if (CHAI_ENABLE_EXAMPLES) add_subdirectory(examples) -endif () +endif() -if (ENABLE_DOCUMENTATION) +if (CHAI_ENABLE_DOCS) add_subdirectory(docs) endif() + +if (CHAI_ENABLE_REPRODUCERS) + add_subdirectory(reproducers) +endif() diff --git a/Dockerfile b/Dockerfile index e690720a..42853e19 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,55 +1,88 @@ -FROM axom/compilers:gcc-5 AS gcc5 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:gcc-6 AS gcc6 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:gcc-7 AS gcc7 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:gcc-8 AS gcc8 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:clang-4 AS clang4 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=clang++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:clang-5 AS clang5 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=clang++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - -FROM axom/compilers:clang-6 AS clang6 -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=clang++ -DENABLE_CUDA=OFF .. -RUN cd build && make -j 16 -RUN cd build && ctest -T test --output-on-failure - - -FROM axom/compilers:nvcc-10 AS nvcc -COPY --chown=axom:axom . /home/axom/workspace -WORKDIR /home/axom/workspace -RUN mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=On .. -RUN cd build && make -j 16 +FROM ghcr.io/rse-ops/gcc-ubuntu-20.04:gcc-7.3.0 AS gcc7 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DENABLE_COVERAGE=On -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/gcc-ubuntu-20.04:gcc-8.1.0 AS gcc8 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/gcc-ubuntu-18.04:gcc-9.4.0 AS gcc9 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/gcc-ubuntu-18.04:gcc-11.2.0 AS gcc11 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/clang-ubuntu-20.04:llvm-10.0.0 AS clang10 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_CXX_COMPILER=clang++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/clang-ubuntu-22.04:llvm-11.0.0 AS clang11 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_CXX_COMPILER=clang++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/clang-ubuntu-22.04:llvm-12.0.0 AS clang12 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_CXX_COMPILER=clang++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/clang-ubuntu-22.04:llvm-13.0.0 AS clang13 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN cmake -DCMAKE_CXX_COMPILER=clang++ .. && \ + make -j 16 && \ + ctest -T test --output-on-failure + +FROM ghcr.io/rse-ops/cuda:cuda-10.1.243-ubuntu-18.04 AS nvcc10 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN . /opt/spack/share/spack/setup-env.sh && spack load cuda && \ + cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=On .. && \ + make -j 16 + +FROM ghcr.io/rse-ops/cuda-ubuntu-20.04:cuda-11.1.1 AS nvcc11 +ENV GTEST_COLOR=1 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN . /opt/spack/share/spack/setup-env.sh && spack load cuda && \ + cmake -DCMAKE_CXX_COMPILER=g++ -DENABLE_CUDA=On .. && \ + make -j 16 + +FROM ghcr.io/rse-ops/hip-ubuntu-22.04:hip-5.1.3 AS hip +ENV GTEST_COLOR=1 +ENV HCC_AMDGPU_TARGET=gfx900 +COPY . /home/chai/workspace +WORKDIR /home/chai/workspace/build +RUN . /opt/spack/share/spack/setup-env.sh && spack load hip llvm-amdgpu && \ + cmake -DENABLE_WARNINGS_AS_ERRORS=Off -DCHAI_ENABLE_MANAGED_PTR=Off -DCMAKE_CXX_COMPILER=amdclang++ -DCMAKE_C_COMPILER=amdclang -DENABLE_HIP=On .. && \ + make -j 16 VERBOSE=1 diff --git a/LICENSE b/LICENSE index 8f8fd45a..86509c8f 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2016-2019, Lawrence Livermore National Security, LLC. +Copyright (c) 2016-2024, Lawrence Livermore National Security, LLC. All rights reserved. Redistribution and use in source and binary forms, with or without diff --git a/README.md b/README.md index e69aee09..44762dc7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,12 @@ -# CHAI v2.4 +[comment]: # (#################################################################) +[comment]: # (Copyright 2016-24, Lawrence Livermore National Security, LLC) +[comment]: # (and CHAI project contributors. See the CHAI LICENSE file for) +[comment]: # (details.) +[comment]: # +[comment]: # (# SPDX-License-Identifier: BSD-3-Clause) +[comment]: # (#################################################################) + +# CHAI v2024.02.0 [![Azure Build Status](https://dev.azure.com/davidbeckingsale/CHAI/_apis/build/status/LLNL.CHAI?branchName=develop)](https://dev.azure.com/davidbeckingsale/CHAI/_build/latest?definitionId=2&branchName=develop) [![Build Status](https://travis-ci.org/LLNL/CHAI.svg?branch=develop)](https://travis-ci.org/LLNL/CHAI) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md new file mode 100644 index 00000000..9dc23c2f --- /dev/null +++ b/RELEASE_NOTES.md @@ -0,0 +1,27 @@ +[comment]: # (#################################################################) +[comment]: # (Copyright 2016-24, Lawrence Livermore National Security, LLC) +[comment]: # (and CHAI project contributors. See the CHAI LICENSE file for) +[comment]: # (details.) +[comment]: # +[comment]: # (# SPDX-License-Identifier: BSD-3-Clause) +[comment]: # (#################################################################) + +# CHAI Software Release Notes + +Notes describing significant changes in each CHAI release are documented +in this file. + +The format of this file is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). + +## [Version 2024.02.0] - Release date 2024-03-04 + +### Added +- Support for APUs with a single memory space. To use, configure with -DCHAI\_DISABLE\_RM=ON -DCHAI\_GPU\_THIN\_ALLOCATE=ON. + +### Changed +- Moved installed CMake targets from share/chai/cmake to lib/cmake/chai to be consistent with other libraries in the RAJA Portability Suite +- Improved dependency handling during the build of CHAI and when it is imported into another library/application +- Removed ArrayManager::enableDeviceSynchronization and ArrayManager::disableDeviceSynchronization. Instead, use the environment variables for device synchronization after all kernels (e.g. CUDA\_LAUNCH\_BLOCKING or HIP\_LAUNCH\_BLOCKING) + +### Fixed +- Use free instead of realloc when the size is 0 (fixes a warning from valgrind) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 903d0a0d..781851f3 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -1,44 +1,45 @@ -strategy: - matrix: - gcc5: - docker_target: gcc5 - gcc6: - docker_target: gcc6 - gcc7: - docker_target: gcc7 - gcc8: - docker_target: gcc8 - clang4: - docker_target: clang4 - clang5: - docker_target: clang5 - clang7: - docker_target: clang6 - nvcc: - docker_target: nvcc - -variables: - DOCKER_BUILDKIT: '1' - -steps: -- checkout: self - clean: boolean - submodules: recursive -- task: Docker@1 - inputs: - command: build - dockerFile: Dockerfile - arguments: '--target $(docker_target)' -- script: | - CID=$(docker create llnl/chai:$(Build.BuildId)) - echo ${CID} - docker cp ${CID}:/home/axom/workspace/build local-build - docker rm ${CID} - displayName: 'Copy test artifacts' - condition: ne( variables['docker_target'], 'nvcc') -- task: PublishTestResults@2 - inputs: - testResultsFormat: 'cTest' - testResultsFiles: '**/Test.xml' - testRunTitle: '$(docker_target) Tests' - condition: ne( variables['docker_target'], 'nvcc') +jobs: +- job: Docker + timeoutInMinutes: 360 + strategy: + matrix: + gcc7: + docker_target: gcc7 + gcc8: + docker_target: gcc8 + gcc9: + docker_target: gcc9 + clang10: + docker_target: clang10 + clang11: + docker_target: clang11 + clang12: + docker_target: clang12 + clang13: + docker_target: clang13 + pool: + vmImage: 'ubuntu-latest' + variables: + DOCKER_BUILDKIT: '1' + steps: + - checkout: self + clean: boolean + submodules: recursive + - task: Docker@1 + inputs: + command: build + dockerFile: 'Dockerfile' + arguments: '--target $(docker_target)' + - script: | + CID=$(docker create llnl/chai:$(Build.BuildId)) + echo ${CID} + docker cp ${CID}:/home/chai/workspace/build local-build + docker rm ${CID} + displayName: 'Copy test artifacts' + condition: ne( variables['docker_target'], 'nvcc') + - task: PublishTestResults@2 + inputs: + testResultsFormat: 'cTest' + testResultsFiles: '**/Test.xml' + testRunTitle: '$(docker_target) Tests' + condition: ne( variables['docker_target'], 'nvcc') diff --git a/benchmarks/CMakeLists.txt b/benchmarks/CMakeLists.txt index 2cb6ec63..f7833707 100644 --- a/benchmarks/CMakeLists.txt +++ b/benchmarks/CMakeLists.txt @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## @@ -8,16 +8,16 @@ set (chai_benchmark_depends chai gbenchmark) -if (ENABLE_CUDA) +if (CHAI_ENABLE_CUDA) set (chai_benchmark_depends ${chai_benchmark_depends} cuda) endif () -if (ENABLE_HIP) +if (CHAI_ENABLE_HIP) set (chai_benchmark_depends ${chai_benchmark_depends} - hip) + blt::hip) endif () blt_add_executable( diff --git a/benchmarks/chai_arraymanager_benchmarks.cpp b/benchmarks/chai_arraymanager_benchmarks.cpp index cd9e7885..4670fa89 100644 --- a/benchmarks/chai_arraymanager_benchmarks.cpp +++ b/benchmarks/chai_arraymanager_benchmarks.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/benchmarks/chai_benchmark_utils.hpp b/benchmarks/chai_benchmark_utils.hpp index 524f1309..73809989 100644 --- a/benchmarks/chai_benchmark_utils.hpp +++ b/benchmarks/chai_benchmark_utils.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/benchmarks/chai_managed_ptr_benchmarks.cpp b/benchmarks/chai_managed_ptr_benchmarks.cpp index 7352f2c5..226fc742 100644 --- a/benchmarks/chai_managed_ptr_benchmarks.cpp +++ b/benchmarks/chai_managed_ptr_benchmarks.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/benchmarks/chai_managedarray_benchmarks.cpp b/benchmarks/chai_managedarray_benchmarks.cpp index 47fb68a1..95834278 100644 --- a/benchmarks/chai_managedarray_benchmarks.cpp +++ b/benchmarks/chai_managedarray_benchmarks.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/blt b/blt index ddd5a0ca..058b312f 160000 --- a/blt +++ b/blt @@ -1 +1 @@ -Subproject commit ddd5a0ca7c566d0ae14270b66625c8a363630ddb +Subproject commit 058b312f8a5ef305e12a4380deaa13d618eff54e diff --git a/cmake/ChaiBasics.cmake b/cmake/ChaiBasics.cmake index a2a96026..f0168158 100644 --- a/cmake/ChaiBasics.cmake +++ b/cmake/ChaiBasics.cmake @@ -1,14 +1,9 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --expt-extended-lambda") -if (ENABLE_HIP) - #bug in ROCm 2.4 is incorrectly warning about missing overrides - set(HIP_HIPCC_FLAGS "${HIP_HIPCC_FLAGS} -Wno-inconsistent-missing-override") -endif() - include(${PROJECT_SOURCE_DIR}/cmake/thirdparty/SetupChaiThirdparty.cmake) diff --git a/cmake/SetupChaiOptions.cmake b/cmake/SetupChaiOptions.cmake new file mode 100644 index 00000000..e4d7414b --- /dev/null +++ b/cmake/SetupChaiOptions.cmake @@ -0,0 +1,46 @@ +############################################################################ +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: BSD-3-Clause +############################################################################ +option(CHAI_ENABLE_GPU_SIMULATION_MODE "Enable GPU Simulation Mode" Off) +option(CHAI_ENABLE_OPENMP "Enable OpenMP" Off) +option(CHAI_ENABLE_MPI "Enable MPI (for umpire replay only)" Off) +option(CHAI_ENABLE_IMPLICIT_CONVERSIONS "Enable implicit conversions to-from raw pointers" On) + +option(CHAI_DISABLE_RM "Make ManagedArray a thin wrapper" Off) +mark_as_advanced(CHAI_DISABLE_RM) + +option(CHAI_ENABLE_UM "Use CUDA unified (managed) memory" Off) +option(CHAI_THIN_GPU_ALLOCATE "Single memory space model" Off) +option(CHAI_ENABLE_PINNED "Use pinned host memory" Off) +option(CHAI_ENABLE_RAJA_PLUGIN "Build plugin to set RAJA execution spaces" Off) +option(CHAI_ENABLE_GPU_ERROR_CHECKING "Enable GPU error checking" On) +option(CHAI_ENABLE_MANAGED_PTR "Enable managed_ptr" On) +option(CHAI_DEBUG "Enable Debug Logging." Off) +option(CHAI_ENABLE_RAJA_NESTED_TEST "Enable raja-chai-nested-tests, which fails to build on Debug CUDA builds." On) +option(CHAI_ENABLE_MANAGED_PTR_ON_GPU "Enable managed_ptr on GPU" On) + +option(CHAI_ENABLE_TESTS "Enable CHAI tests" On) +option(CHAI_ENABLE_BENCHMARKS "Enable benchmarks" On) +option(CHAI_ENABLE_EXAMPLES "Enable CHAI examples" On) +option(CHAI_ENABLE_REPRODUCERS "Enable CHAI reproducers" Off) +option(CHAI_ENABLE_DOCS "Enable CHAI documentation" Off) + +# options for Umpire as TPL +set(ENABLE_GMOCK On CACHE BOOL "") +option(CHAI_ENABLE_ASSERTS "Build Umpire with assert() enabled" On) +set(ENABLE_GTEST_DEATH_TESTS ${CHAI_ENABLE_ASSERTS} CACHE BOOL "") + +option(CHAI_ENABLE_COPY_HEADERS "Enable CHAI copy headers" Off) + +set(ENABLE_CUDA Off CACHE BOOL "Enable CUDA") + +if (CHAI_ENABLE_UM AND NOT ENABLE_CUDA AND NOT CHAI_THIN_GPU_ALLOCATE) + message(FATAL_ERROR "Option CHAI_ENABLE_UM requires ENABLE_CUDA or CHAI_THIN_GPU_ALLOCATE") +endif() + +if (CHAI_THIN_GPU_ALLOCATE AND NOT CHAI_DISABLE_RM) + message(FATAL_ERROR "Option CHAI_THIN_GPU_ALLOCATE requires CHAI_DISABLE_RM") +endif() diff --git a/cmake/chai-config.cmake.in b/cmake/chai-config.cmake.in new file mode 100644 index 00000000..525ea610 --- /dev/null +++ b/cmake/chai-config.cmake.in @@ -0,0 +1,33 @@ +############################################################################## +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the COPYRIGHT file for details. +# +# SPDX-License-Identifier: BSD-3-Clause +############################################################################## + +@PACKAGE_INIT@ + +# cache the prefix dir (could be overriden by find_dependency) +set(CHAI_PACKAGE_PREFIX_DIR ${PACKAGE_PREFIX_DIR}) + +include(CMakeFindDependencyMacro) + +include("${CMAKE_CURRENT_LIST_DIR}/BLTSetupTargets.cmake") + +if (NOT TARGET umpire) + find_dependency(umpire CONFIG NO_DEFAULT_PATH PATHS + ${umpire_DIR} + @umpire_DIR@ + ${CHAI_PACKAGE_PREFIX_DIR}) +endif () + +if (@CHAI_ENABLE_RAJA_PLUGIN@ AND NOT TARGET RAJA) + find_dependency(raja CONFIG NO_DEFAULT_PATH PATHS + ${raja_DIR} + @raja_DIR@ + ${CHAI_PACKAGE_PREFIX_DIR}) +endif () + +include("${CMAKE_CURRENT_LIST_DIR}/chai-targets.cmake") + +check_required_components(chai) diff --git a/cmake/thirdparty/SetupChaiThirdparty.cmake b/cmake/thirdparty/SetupChaiThirdparty.cmake index 13c2c23a..f42d949d 100644 --- a/cmake/thirdparty/SetupChaiThirdparty.cmake +++ b/cmake/thirdparty/SetupChaiThirdparty.cmake @@ -1,69 +1,42 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## -if (NOT TARGET umpire) - if (DEFINED umpire_DIR) - # this allows umpire_DIR to be the install prefix if we are relying on umpire's - # installed umpire-config.cmake - list(APPEND CMAKE_PREFIX_PATH ${umpire_DIR}) - find_package(umpire REQUIRED) - - if (ENABLE_MPI) - set(UMPIRE_DEPENDS mpi) - else() - set(UMPIRE_DEPENDS) - endif() - blt_register_library( - NAME umpire - INCLUDES ${UMPIRE_INCLUDE_DIRS} - LIBRARIES umpire - DEPENDS_ON ${UMPIRE_DEPENDS}) +if (NOT TARGET umpire) + if (DEFINED umpire_DIR OR DEFINED UMPIRE_DIR) + message(STATUS "[CHAI] Using external Umpire") + find_package(umpire CONFIG REQUIRED NO_DEFAULT_PATH PATHS ${umpire_DIR} ${UMPIRE_DIR}) else () - set(OLD_ENABLE_FORTRAN ${ENABLE_FORTRAN}) - set(ENABLE_FORTRAN Off CACHE BOOL "Enable Fortran in Umpire") - add_subdirectory(${PROJECT_SOURCE_DIR}/src/tpl/umpire) - set(ENABLE_FORTRAN ${OLD_ENABLE_FORTRAN}) - endif() - - # Umpire depends on camp - if (NOT TARGET camp) - if (DEFINED camp_DIR) - find_package(camp REQUIRED) - set_target_properties(camp PROPERTIES IMPORTED_GLOBAL TRUE) + if (NOT EXISTS ${PROJECT_SOURCE_DIR}/src/tpl/umpire/CMakeLists.txt) + message(FATAL_ERROR "[CHAI] Umpire not found! Set umpire_DIR to the install location of Umpire or run 'git submodule update --init --recursive' in the CHAI repository, then try building again.") else () - message(FATAL_ERROR "camp is required. Please set camp_DIR") - endif() + message(STATUS "[CHAI] Using internal Umpire") - if(ENABLE_CUDA) - blt_add_target_definitions( - TO camp - SCOPE INTERFACE - TARGET_DEFINITIONS CAMP_HAVE_CUDA) - endif() + set(UMPIRE_ENABLE_FORTRAN Off CACHE BOOL "Enable Fortran in Umpire") + set(UMPIRE_ENABLE_C Off CACHE BOOL "Enable Fortran in Umpire") + set(UMPIRE_ENABLE_TESTS Off CACHE BOOL "Enable tests in Umpire") + set(UMPIRE_ENABLE_TOOLS Off CACHE BOOL "Enable tools in Umpire") + + add_subdirectory(${PROJECT_SOURCE_DIR}/src/tpl/umpire) + endif () endif () -endif() +endif () -if (ENABLE_RAJA_PLUGIN) +if (CHAI_ENABLE_RAJA_PLUGIN) if (NOT TARGET RAJA) - if (DEFINED RAJA_DIR) - # this allows RAJA_DIR to be the install prefix if we are relying on RAJA's - # installed RAJA-config.cmake - list(APPEND CMAKE_PREFIX_PATH ${RAJA_DIR}) - message(STATUS "CHAI: using external RAJA via find_package ${RAJA_DIR}") - find_package(RAJA REQUIRED) - blt_register_library( - NAME RAJA - INCLUDES ${RAJA_INCLUDE_DIRS} - LIBRARIES RAJA) - else() - message(STATUS "CHAI: using builtin RAJA submodule") - add_subdirectory(${PROJECT_SOURCE_DIR}/src/tpl/raja) - endif() - else() - message(STATUS "CHAI: using existing RAJA target") - endif() + if (DEFINED raja_DIR OR DEFINED RAJA_DIR) + message(STATUS "[CHAI] Using external RAJA") + find_package(raja CONFIG REQUIRED NO_DEFAULT_PATH PATHS ${raja_DIR} ${RAJA_DIR}) + else () + if (NOT EXISTS ${PROJECT_SOURCE_DIR}/src/tpl/raja/CMakeLists.txt) + message(FATAL_ERROR "[CHAI] RAJA not found! Set raja_DIR to the install location of RAJA or run 'git submodule update --init --recursive' in the CHAI repository, then try building again.") + else () + message(STATUS "[CHAI] Using internal RAJA") + add_subdirectory(${PROJECT_SOURCE_DIR}/src/tpl/raja) + endif () + endif () + endif () endif () diff --git a/docs/CMakeLists.txt b/docs/CMakeLists.txt index 34a2ced2..338cdfcd 100644 --- a/docs/CMakeLists.txt +++ b/docs/CMakeLists.txt @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/docs/doxygen/CMakeLists.txt b/docs/doxygen/CMakeLists.txt index c53655c2..dee5911a 100644 --- a/docs/doxygen/CMakeLists.txt +++ b/docs/doxygen/CMakeLists.txt @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/docs/doxygen/Doxyfile.in b/docs/doxygen/Doxyfile.in index 64c68ce8..1646cd60 100644 --- a/docs/doxygen/Doxyfile.in +++ b/docs/doxygen/Doxyfile.in @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## @@ -338,10 +338,10 @@ GENERATE_BUGLIST = YES GENERATE_DEPRECATEDLIST= YES -# The ENABLED_SECTIONS tag can be used to enable conditional +# The CHAI_ENABLED_SECTIONS tag can be used to enable conditional # documentation sections, marked by \if sectionname ... \endif. -ENABLED_SECTIONS = +CHAI_ENABLED_SECTIONS = # The MAX_INITIALIZER_LINES tag determines the maximum number of lines # the initial value of a variable or define consists of for it to appear in @@ -973,11 +973,11 @@ PERLMOD_MAKEVAR_PREFIX = # Configuration options related to the preprocessor #--------------------------------------------------------------------------- -# If the ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will +# If the CHAI_ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will # evaluate all C-preprocessor directives found in the sources and include # files. -ENABLE_PREPROCESSING = YES +CHAI_ENABLE_PREPROCESSING = YES # If the MACRO_EXPANSION tag is set to YES Doxygen will expand all macro # names in the source code. If set to NO (the default) only conditional @@ -1139,14 +1139,14 @@ UML_LOOK = NO TEMPLATE_RELATIONS = YES -# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT +# If the CHAI_ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT # tags are set to YES then doxygen will generate a graph for each documented # file showing the direct and indirect include dependencies of the file with # other documented files. INCLUDE_GRAPH = NO -# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and +# If the CHAI_ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and # HAVE_DOT tags are set to YES then doxygen will generate a graph for each # documented header file showing the documented files that directly or # indirectly include this file. diff --git a/docs/sphinx/CMakeLists.txt b/docs/sphinx/CMakeLists.txt index acc991ab..f819c905 100644 --- a/docs/sphinx/CMakeLists.txt +++ b/docs/sphinx/CMakeLists.txt @@ -1,50 +1,8 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## -####################################################################### -# Copyright (c) 2016-2018, Lawrence Livermore National Security, LLC. All -# rights reserved. -# -# Produced at the Lawrence Livermore National Laboratory -# -# This file is part of CHAI. -# -# LLNL-CODE-705877 -# -# For details, see https:://github.com/LLNL/CHAI -# Please also see the NOTICE and LICENSE files. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# -# - Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# -# - Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the -# distribution. -# -# - Neither the name of the LLNS/LLNL nor the names of its contributors -# may be used to endorse or promote products derived from this -# software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, -# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, -# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS -# OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED -# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY -# WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE -# POSSIBILITY OF SUCH DAMAGE. -####################################################################### blt_add_sphinx_target(chai_docs) diff --git a/docs/sphinx/advanced_configuration.rst b/docs/sphinx/advanced_configuration.rst index d86344f2..846f5048 100644 --- a/docs/sphinx/advanced_configuration.rst +++ b/docs/sphinx/advanced_configuration.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _advanced_configuration: ================ @@ -17,10 +23,10 @@ Here is a summary of the configuration options, their default value, and meaning =========================== ======== =============================================================================== ENABLE_CUDA Off Enable CUDA support. ENABLE_HIP Off Enable HIP support. - ENABLE_GPU_SIMULATION_MODE Off Simulates GPU execution. - ENABLE_UM Off Enable support for CUDA Unified Memory. - ENABLE_IMPLICIT_CONVERSIONS On Enable implicit conversions between ManagedArray and raw pointers - DISABLE_RM Off Disable the ArrayManager and make ManagedArray a thin wrapper around a pointer. + CHAI_ENABLE_GPU_SIMULATION_MODE Off Simulates GPU execution. + CHAI_ENABLE_UM Off Enable support for CUDA Unified Memory. + CHAI_ENABLE_IMPLICIT_CONVERSIONS On Enable implicit conversions between ManagedArray and raw pointers + CHAI_DISABLE_RM Off Disable the ArrayManager and make ManagedArray a thin wrapper around a pointer. ENABLE_TESTS On Build test executables. ENABLE_BENCHMARKS On Build benchmark programs. =========================== ======== =============================================================================== @@ -35,23 +41,23 @@ These arguments are explained in more detail below: This option enables support for GPUs using HIP. If CHAI is built without CUDA, HIP, or GPU_SIMULATION_MODE support, then only the ``CPU`` execution space is available for use. -* ENABLE_GPU_SIMULATION_MODE +* CHAI_ENABLE_GPU_SIMULATION_MODE This option simulates GPU support by enabling the GPU execution space, backed by a HOST umpire allocator. If CHAI is built without CUDA, HIP, or GPU_SIMULATION_MODE support, then only the ``CPU`` execution space is available for use. -* ENABLE_UM +* CHAI_ENABLE_UM This option enables support for Unified Memory as an optional execution space. When a ``ManagedArray`` is allocated in the ``UM`` space, CHAI will not manually copy data. Data movement in this case is handled by the CUDA driver and runtime. -* ENABLE_IMPLICIT_CONVERSIONS +* CHAI_ENABLE_IMPLICIT_CONVERSIONS This option will allow implicit casting between an object of type ``ManagedArray`` and the correpsonding raw pointer type ``T*``. This option is disabled by default, and should be used with caution. -* DISABLE_RM +* CHAI_DISABLE_RM This option will remove all usage of the ``ArrayManager`` class and let the ``ManagedArray`` objects function as thin wrappers around a raw pointer. This option can be used with CPU-only allocations, or with CUDA Unified Memory. diff --git a/docs/sphinx/code_documentation.rst b/docs/sphinx/code_documentation.rst index 575e567b..b17c93f1 100644 --- a/docs/sphinx/code_documentation.rst +++ b/docs/sphinx/code_documentation.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _code_documentation: ================== diff --git a/docs/sphinx/conf.py b/docs/sphinx/conf.py index 8c3bb894..eeba20ee 100644 --- a/docs/sphinx/conf.py +++ b/docs/sphinx/conf.py @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## @@ -61,9 +61,9 @@ # built documents. # # The short X.Y version. -version = u'2.4' +version = u'2024.02' # The full version, including alpha/beta/rc tags. -release = u'2.4.0' +release = u'2024.02.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/sphinx/conf.py.in b/docs/sphinx/conf.py.in index 9b08a122..6905e91f 100644 --- a/docs/sphinx/conf.py.in +++ b/docs/sphinx/conf.py.in @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## @@ -60,9 +60,9 @@ author = u'' # built documents. # # The short X.Y version. -version = u'2.4' +version = u'2024.02' # The full version, including alpha/beta/rc tags. -release = u'2.4.0' +release = u'2024.02.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/sphinx/contribution_guide.rst b/docs/sphinx/contribution_guide.rst index 9bf3cfc1..cdcbf190 100644 --- a/docs/sphinx/contribution_guide.rst +++ b/docs/sphinx/contribution_guide.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _contribution_guide: ================== diff --git a/docs/sphinx/developer/ci.rst b/docs/sphinx/developer/ci.rst index 7c3c787b..be42733b 100644 --- a/docs/sphinx/developer/ci.rst +++ b/docs/sphinx/developer/ci.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _ci: ====================== diff --git a/docs/sphinx/developer/uberenv.rst b/docs/sphinx/developer/uberenv.rst index 03857e63..0c8440e3 100644 --- a/docs/sphinx/developer/uberenv.rst +++ b/docs/sphinx/developer/uberenv.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _developer_guide: =============== @@ -33,20 +39,20 @@ Vetted specs $ ls -c1 .gitlab/*jobs.yml .gitlab/lassen-jobs.yml - .gitlab/quartz-jobs.yml + .gitlab/ruby-jobs.yml -CI contains jobs for quartz. +CI contains jobs for ruby. .. code-block:: bash - $ git grep -h "SPEC" .gitlab/quartz-jobs.yml | grep "gcc" + $ git grep -h "SPEC" .gitlab/ruby-jobs.yml | grep "gcc" SPEC: "%gcc@4.9.3" SPEC: "%gcc@6.1.0" SPEC: "%gcc@7.1.0" SPEC: "%gcc@7.3.0" SPEC: "%gcc@8.1.0" -We now have a list of the specs vetted on ``quartz``/``toss_3_x86_64_ib``. +We now have a list of the specs vetted on ``ruby``/``toss_3_x86_64_ib``. .. note:: In practice, one should check if the job is not *allowed to fail*, or even deactivated. @@ -60,7 +66,7 @@ In CHAI, the Spack configuration for MacOS contains the default compilers depend Using Uberenv to generate the host-config file ---------------------------------------------- -We have seen that we can safely use `gcc@8.1.0` on quartz. Let us ask for the default configuration first, and then ask for RAJA support and link to develop version of RAJA: +We have seen that we can safely use `gcc@8.1.0` on ruby. Let us ask for the default configuration first, and then ask for RAJA support and link to develop version of RAJA: .. code-block:: bash @@ -71,7 +77,7 @@ Each will generate a CMake cache file, e.g.: .. code-block:: bash - hc-quartz-toss_3_x86_64_ib-clang@9.0.0-fjcjwd6ec3uen5rh6msdqujydsj74ubf.cmake + hc-ruby-toss_3_x86_64_ib-clang@9.0.0-fjcjwd6ec3uen5rh6msdqujydsj74ubf.cmake Using host-config files to build CHAI ------------------------------------- diff --git a/docs/sphinx/developer_guide.rst b/docs/sphinx/developer_guide.rst index 81a5648b..615e2191 100644 --- a/docs/sphinx/developer_guide.rst +++ b/docs/sphinx/developer_guide.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _developer_guide: =============== diff --git a/docs/sphinx/getting_started.rst b/docs/sphinx/getting_started.rst index 10600ca2..a0aa7e4b 100644 --- a/docs/sphinx/getting_started.rst +++ b/docs/sphinx/getting_started.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _getting_started: =============== diff --git a/docs/sphinx/index.rst b/docs/sphinx/index.rst index b391a334..94e3a584 100644 --- a/docs/sphinx/index.rst +++ b/docs/sphinx/index.rst @@ -1,43 +1,8 @@ -.. Copyright (c) 2016, Lawrence Livermore National Security, LLC. All - rights reserved. - - Produced at the Lawrence Livermore National Laboratory - - This file is part of CHAI. - - LLNL-CODE-705877 - - For details, see https:://github.com/LLNL/CHAI - Please also see the NOTICE and LICENSE files. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - - Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - - Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the - distribution. - - - Neither the name of the LLNS/LLNL nor the names of its contributors - may be used to endorse or promote products derived from this - software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS - OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED - AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY - WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause ****** CHAI diff --git a/docs/sphinx/tutorial.rst b/docs/sphinx/tutorial.rst index 5818327e..1aa441d6 100644 --- a/docs/sphinx/tutorial.rst +++ b/docs/sphinx/tutorial.rst @@ -1,3 +1,9 @@ +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause + .. _tutorial: ======== diff --git a/docs/sphinx/user_guide.rst b/docs/sphinx/user_guide.rst index f28e7917..85817b25 100644 --- a/docs/sphinx/user_guide.rst +++ b/docs/sphinx/user_guide.rst @@ -1,43 +1,8 @@ -.. Copyright (c) 2016, Lawrence Livermore National Security, LLC. All - rights reserved. - - Produced at the Lawrence Livermore National Laboratory - - This file is part of CHAI. - - LLNL-CODE-705877 - - For details, see https:://github.com/LLNL/CHAI - Please also see the NOTICE and LICENSE files. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - - Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - - Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the - distribution. - - - Neither the name of the LLNS/LLNL nor the names of its contributors - may be used to endorse or promote products derived from this - software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS - OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED - AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY - WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. +.. + # Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI + # project contributors. See the CHAI LICENSE file for details. + # + # SPDX-License-Identifier: BSD-3-Clause .. _user_guide: diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 09397813..fef82b96 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -1,21 +1,21 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## set (chai_umpire_example_depends chai) -if (ENABLE_CUDA) +if (CHAI_ENABLE_CUDA) set (chai_umpire_example_depends ${chai_umpire_example_depends} cuda) endif() -if (ENABLE_HIP) +if (CHAI_ENABLE_HIP) set (chai_umpire_example_depends ${chai_umpire_example_depends} - hip) + blt::hip) endif() blt_add_executable( @@ -23,7 +23,7 @@ blt_add_executable( SOURCES chai-umpire-allocators.cpp DEPENDS_ON ${chai_umpire_example_depends}) -if (ENABLE_CUDA OR ENABLE_HIP) +if (CHAI_ENABLE_CUDA OR CHAI_ENABLE_HIP) blt_add_executable( NAME chai-example.exe SOURCES example.cpp @@ -35,3 +35,10 @@ if (ENABLE_CUDA OR ENABLE_HIP) DEPENDS_ON ${chai_umpire_example_depends}) endif () +if (CHAI_ENABLE_MANAGED_PTR) + blt_add_executable( + NAME managed_ptr_example.exe + SOURCES managed_ptr_example.cpp + DEPENDS_ON ${chai_umpire_example_depends}) +endif () + diff --git a/examples/chai-umpire-allocators.cpp b/examples/chai-umpire-allocators.cpp index 2e82c6c5..63c2c8e2 100644 --- a/examples/chai-umpire-allocators.cpp +++ b/examples/chai-umpire-allocators.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/examples/ex1.cpp b/examples/ex1.cpp index 39a75e5d..ab7d0d29 100644 --- a/examples/ex1.cpp +++ b/examples/ex1.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/examples/example.cpp b/examples/example.cpp index 33f84695..eeb0d7db 100644 --- a/examples/example.cpp +++ b/examples/example.cpp @@ -1,51 +1,9 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// -// --------------------------------------------------------------------- -// Copyright (c) 2016-2018, Lawrence Livermore National Security, LLC. All -// rights reserved. -// -// Produced at the Lawrence Livermore National Laboratory. -// -// This file is part of CHAI. -// -// LLNL-CODE-705877 -// -// For details, see https:://github.com/LLNL/CHAI -// Please also see the NOTICE and LICENSE files. -// -// Redistribution and use in source and binary forms, with or without -// modification, are permitted provided that the following conditions -// are met: -// -// - Redistributions of source code must retain the above copyright -// notice, this list of conditions and the following disclaimer. -// -// - Redistributions in binary form must reproduce the above copyright -// notice, this list of conditions and the following disclaimer in the -// documentation and/or other materials provided with the -// distribution. -// -// - Neither the name of the LLNS/LLNL nor the names of its contributors -// may be used to endorse or promote products derived from this -// software without specific prior written permission. -// -// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -// HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, -// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, -// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS -// OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED -// AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY -// WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE -// POSSIBILITY OF SUCH DAMAGE. -// --------------------------------------------------------------------- #include "../src/util/forall.hpp" #include "chai/ManagedArray.hpp" diff --git a/examples/managed_ptr_example.cpp b/examples/managed_ptr_example.cpp new file mode 100644 index 00000000..e65081a1 --- /dev/null +++ b/examples/managed_ptr_example.cpp @@ -0,0 +1,52 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: BSD-3-Clause +////////////////////////////////////////////////////////////////////////////// +#include "chai/managed_ptr.hpp" +#include "../src/util/forall.hpp" +#include + +class TestBase { + public: + CHAI_HOST_DEVICE TestBase() {} + CHAI_HOST_DEVICE virtual ~TestBase() {} + CHAI_HOST_DEVICE virtual int getValue() const = 0; +}; + +class TestDerived : public TestBase { + public: + CHAI_HOST_DEVICE TestDerived(const int value) : TestBase(), m_value(value) {} + CHAI_HOST_DEVICE virtual ~TestDerived() {} + CHAI_HOST_DEVICE virtual int getValue() const { return m_value; } + + private: + int m_value; +}; + +int main(int CHAI_UNUSED_ARG(argc), char** CHAI_UNUSED_ARG(argv)) +{ + // Create an object on the host, and if either CUDA or HIP is enabled, + // on the device as well + chai::managed_ptr derived = chai::make_managed(42); + + // chai::managed_ptr can be accessed on the host + forall(sequential(), 0, 1, [=] CHAI_HOST (int) { + printf("Result of virtual function call on host: %d\n", derived->getValue()); + }); + +#if defined(CHAI_GPUCC) + // chai::managed_ptr can be accessed on the device + forall(gpu(), 0, 1, [=] CHAI_DEVICE (int) { + printf("Result of virtual function call on device: %d\n", derived->getValue()); + }); +#endif + + // Free the object on the host, and if either CUDA or HIP is enabled, + // on the device as well + derived.free(); + + return 0; +} + diff --git a/examples/pinned.cpp b/examples/pinned.cpp index c819b725..dccd2ef3 100644 --- a/examples/pinned.cpp +++ b/examples/pinned.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/reproducers/CMakeLists.txt b/reproducers/CMakeLists.txt new file mode 100644 index 00000000..e5ec4348 --- /dev/null +++ b/reproducers/CMakeLists.txt @@ -0,0 +1,33 @@ +############################################################################## +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: BSD-3-Clause +############################################################################## +set (chai_reproducer_depends chai) + +if (CHAI_ENABLE_CUDA) + list(APPEND chai_reproducer_depends cuda) +endif () + +if (CHAI_ENABLE_HIP) + list(APPEND chai_reproducer_depends blt::hip) +endif () + +if (CHAI_ENABLE_MANAGED_PTR AND CHAI_ENABLE_HIP) + blt_add_executable( + NAME virtual_function_simple_reproducer.exe + SOURCES virtual_function_simple_reproducer.cpp + DEPENDS_ON ${chai_reproducer_depends}) + + blt_add_executable( + NAME virtual_function_complex_reproducer.exe + SOURCES virtual_function_complex_reproducer.cpp + DEPENDS_ON ${chai_reproducer_depends}) + + blt_add_executable( + NAME managed_ptr_reproducer.exe + SOURCES managed_ptr_reproducer.cpp + DEPENDS_ON ${chai_reproducer_depends}) +endif () + diff --git a/reproducers/managed_ptr_reproducer.cpp b/reproducers/managed_ptr_reproducer.cpp new file mode 100644 index 00000000..67255818 --- /dev/null +++ b/reproducers/managed_ptr_reproducer.cpp @@ -0,0 +1,54 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: BSD-3-Clause +////////////////////////////////////////////////////////////////////////////// +#include "chai/managed_ptr.hpp" +#include +#include +#include +#include + +class TestBase { + public: + __host__ __device__ TestBase() {} + __host__ __device__ virtual ~TestBase() {} + __host__ __device__ virtual int getValue() const = 0; +}; + +class TestDerived : public TestBase { + public: + __host__ __device__ TestDerived(const int value) : TestBase(), m_value(value) {} + __host__ __device__ virtual ~TestDerived() {} + __host__ __device__ virtual int getValue() const { return m_value; } + + private: + int m_value; +}; + +__global__ void callDeviceVirtualFunction(int* d_result, + chai::managed_ptr derived) { + *d_result = derived->getValue(); +} + +int main(int, char**) { + chai::managed_ptr derived = chai::make_managed(42); + + int* d_result; + hipMalloc(&d_result, sizeof(int)); + + hipLaunchKernelGGL(callDeviceVirtualFunction, 1, 1, 0, 0, d_result, derived); + + int* h_result = (int*) malloc(sizeof(int)); + hipMemcpyDtoH(h_result, d_result, sizeof(int)); + + printf("Result: %d\n", *h_result); + + free(h_result); + hipFree(d_result); + derived.free(); + + return 0; +} + diff --git a/reproducers/virtual_function_complex_reproducer.cpp b/reproducers/virtual_function_complex_reproducer.cpp new file mode 100644 index 00000000..ee70ed07 --- /dev/null +++ b/reproducers/virtual_function_complex_reproducer.cpp @@ -0,0 +1,84 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: BSD-3-Clause +////////////////////////////////////////////////////////////////////////////// +#include "chai/managed_ptr.hpp" +#include +#include +#include +#include + +class TestBase { + public: + __host__ __device__ TestBase() {} + __host__ __device__ virtual ~TestBase() {} + __host__ __device__ virtual int getValue() const = 0; +}; + +class TestDerived : public TestBase { + public: + __host__ __device__ TestDerived(const int value) : TestBase(), m_value(value) {} + __host__ __device__ virtual ~TestDerived() {} + __host__ __device__ virtual int getValue() const { return m_value; } + + private: + int m_value; +}; + +__global__ void createOnDevice(TestDerived** ptr, int val) { + *ptr = new TestDerived(val); +} + +__global__ void destroyOnDevice(TestBase* ptr) { + delete ptr; +} + +__global__ void callDeviceVirtualFunction(int* d_result, TestBase* derived) { + *d_result = derived->getValue(); +} + +int main(int, char**) { + // Allocate space on the GPU to hold the pointer to the new object + TestDerived** gpuBuffer; + hipMalloc((void**)(&gpuBuffer), sizeof(TestDerived*)); + + // New the object on the device + hipLaunchKernelGGL(createOnDevice, 1, 1, 0, 0, gpuBuffer, 42); + + // Allocate space on the CPU for the GPU pointer and copy the pointer to the CPU + TestDerived** cpuBuffer = (TestDerived**) malloc(sizeof(TestDerived*)); + hipMemcpyDtoH(cpuBuffer, gpuBuffer, sizeof(TestDerived*)); + + // Extract the pointer to the object on the GPU + TestBase* derived = *cpuBuffer; + + // Free the host and device buffers + free(cpuBuffer); + hipFree(gpuBuffer); + + // Allocate space to hold the result of calling a virtual function on the device + int* d_result; + hipMalloc(&d_result, sizeof(int)); + + // Call a virtual function on the device + hipLaunchKernelGGL(callDeviceVirtualFunction, 1, 1, 0, 0, d_result, derived); + + // Copy the result to the CPU + int* h_result = (int*) malloc(sizeof(int)); + hipMemcpyDtoH(h_result, d_result, sizeof(int)); + + // Print the result + printf("Result: %d\n", *h_result); + + // Free memory + free(h_result); + hipFree(d_result); + + // Delete the object on the device + hipLaunchKernelGGL(destroyOnDevice, 1, 1, 0, 0, derived); + + return 0; +} + diff --git a/reproducers/virtual_function_simple_reproducer.cpp b/reproducers/virtual_function_simple_reproducer.cpp new file mode 100644 index 00000000..7bbb8587 --- /dev/null +++ b/reproducers/virtual_function_simple_reproducer.cpp @@ -0,0 +1,41 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: BSD-3-Clause +////////////////////////////////////////////////////////////////////////////// +#include "chai/managed_ptr.hpp" +#include +#include +#include +#include + +class TestBase { + public: + __host__ __device__ TestBase() {} + __host__ __device__ virtual ~TestBase() {} + __host__ __device__ virtual int getValue() const = 0; +}; + +class TestDerived : public TestBase { + public: + __host__ __device__ TestDerived(const int value) : TestBase(), m_value(value) {} + __host__ __device__ virtual ~TestDerived() {} + __host__ __device__ virtual int getValue() const { return m_value; } + + private: + int m_value; +}; + +__global__ void callVirtualFunction() { + TestBase* derived = new TestDerived(42); + int result = derived->getValue(); + (void) result; + return; +} + +int main(int, char**) { + hipLaunchKernelGGL(callVirtualFunction, 1, 1, 0, 0); + return 0; +} + diff --git a/scripts/apply-license-info.sh b/scripts/apply-license-info.sh index cb56e08c..1a52eb3f 100755 --- a/scripts/apply-license-info.sh +++ b/scripts/apply-license-info.sh @@ -1,7 +1,7 @@ #!/usr/bin/env zsh ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/scripts/check-license-info.sh b/scripts/check-license-info.sh new file mode 100755 index 00000000..9dfae9c1 --- /dev/null +++ b/scripts/check-license-info.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env zsh +############################################################################## +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## + +# This is used for the ~*tpl* line to ignore files in bundled tpls +setopt extended_glob + +autoload colors + +RED="\033[1;31m" +GREEN="\033[1;32m" +NOCOLOR="\033[0m" + +files_no_license=$(grep -rL "the CHAI LICENSE file" . \ + --exclude-dir=.git \ + --exclude-dir=blt \ + --exclude-dir=umpire \ + --exclude-dir=raja \ + --exclude-dir=radiuss-spack-configs \ + --exclude-dir=uberenv) + +if [ $files_no_license ]; then + print "${RED} [!] Some files are missing license text: ${NOCOLOR}" + echo "${files_no_license}" + exit 255 +else + print "${GREEN} [Ok] All files have required license info." + exit 0 +fi diff --git a/scripts/format-source.sh b/scripts/format-source.sh index 91081e1a..7e1d3c0b 100755 --- a/scripts/format-source.sh +++ b/scripts/format-source.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/scripts/gitlab/build_and_test.sh b/scripts/gitlab/build_and_test.sh index 072f606b..2f2466f5 100755 --- a/scripts/gitlab/build_and_test.sh +++ b/scripts/gitlab/build_and_test.sh @@ -1,26 +1,69 @@ #!/usr/bin/env bash + +# Initialize modules for users not using bash as a default shell +if test -e /usr/share/lmod/lmod/init/bash +then + . /usr/share/lmod/lmod/init/bash +fi + ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## + set -o errexit set -o nounset option=${1:-""} hostname="$(hostname)" +truehostname=${hostname//[0-9]/} project_dir="$(pwd)" -build_root=${BUILD_ROOT:-""} hostconfig=${HOST_CONFIG:-""} spec=${SPEC:-""} +module_list=${MODULE_LIST:-""} job_unique_id=${CI_JOB_ID:-""} +use_dev_shm=${USE_DEV_SHM:-true} raja_version=${UPDATE_RAJA:-""} umpire_version=${UPDATE_UMPIRE:-""} +if [[ -n ${module_list} ]] +then + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ Modules to load: ${module_list}" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + module load ${module_list} +fi + +prefix="" + +if [[ -d /dev/shm && ${use_dev_shm} == true ]] +then + prefix="/dev/shm/${hostname}" + if [[ -z ${job_unique_id} ]]; then + job_unique_id=manual_job_$(date +%s) + while [[ -d ${prefix}-${job_unique_id} ]] ; do + sleep 1 + job_unique_id=manual_job_$(date +%s) + done + fi + + prefix="${prefix}-${job_unique_id}" + mkdir -p ${prefix} +else + # We set the prefix in the parent directory so that spack dependencies are not installed inside the source tree. + prefix="$(pwd)/../spack-and-build-root" + mkdir -p ${prefix} +fi + # Dependencies +date +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +echo "~~~~~ Build and test started" +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" if [[ "${option}" != "--build-only" && "${option}" != "--test-only" ]] then echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" @@ -29,7 +72,7 @@ then if [[ -z ${spec} ]] then - echo "SPEC is undefined, aborting..." + echo "[Error]: SPEC is undefined, aborting..." exit 1 fi @@ -50,45 +93,41 @@ then [[ -n ${extra_variants} ]] && spec="${spec} ${extra_variants}" [[ -n ${extra_deps} ]] && spec="${spec} ${extra_deps}" - prefix_opt="" + prefix_opt="--prefix=${prefix}" - if [[ -d /dev/shm ]] - then - prefix="/dev/shm/${hostname}" - if [[ -z ${job_unique_id} ]]; then - job_unique_id=manual_job_$(date +%s) - while [[ -d ${prefix}/${job_unique_id} ]] ; do - sleep 1 - job_unique_id=manual_job_$(date +%s) - done - fi - - prefix="${prefix}/${job_unique_id}" - mkdir -p ${prefix} - prefix_opt="--prefix=${prefix}" - fi + # We force Spack to put all generated files (cache and configuration of + # all sorts) in a unique location so that there can be no collision + # with existing or concurrent Spack. + spack_user_cache="${prefix}/spack-user-cache" + export SPACK_DISABLE_LOCAL_CONFIG="" + export SPACK_USER_CACHE_PATH="${spack_user_cache}" + mkdir -p ${spack_user_cache} - python scripts/uberenv/uberenv.py --spec="${spec}" ${prefix_opt} + ./scripts/uberenv/uberenv.py --spec="${spec}" ${prefix_opt} fi + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ Dependencies Built" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +date # Host config file if [[ -z ${hostconfig} ]] then # If no host config file was provided, we assume it was generated. # This means we are looking of a unique one in project dir. - hostconfigs=( $( ls "${project_dir}/"hc-*.cmake ) ) + hostconfigs=( $( ls "${project_dir}/"*.cmake ) ) if [[ ${#hostconfigs[@]} == 1 ]] then hostconfig_path=${hostconfigs[0]} echo "Found host config file: ${hostconfig_path}" elif [[ ${#hostconfigs[@]} == 0 ]] then - echo "No result for: ${project_dir}/hc-*.cmake" + echo "No result for: ${project_dir}/*.cmake" echo "Spack generated host-config not found." exit 1 else - echo "More than one result for: ${project_dir}/hc-*.cmake" + echo "More than one result for: ${project_dir}/*.cmake" echo "${hostconfigs[@]}" echo "Please specify one with HOST_CONFIG variable" exit 1 @@ -98,65 +137,134 @@ else hostconfig_path="${project_dir}/host-configs/${hostconfig}" fi +hostconfig=$(basename ${hostconfig_path}) + # Build Directory -if [[ -z ${build_root} ]] -then - build_root=$(pwd) -fi +# When using /dev/shm, we use prefix for both spack builds and source build, unless BUILD_ROOT was defined +build_root=${BUILD_ROOT:-"${prefix}"} build_dir="${build_root}/build_${hostconfig//.cmake/}" +install_dir="${build_root}/install_${hostconfig//.cmake/}" -echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" -echo "~~~~~ Host-config: ${hostconfig_path}" -echo "~~~~~ Build Dir: ${build_dir}" -echo "~~~~~ Project Dir: ${project_dir}" -echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +cmake_exe=`grep 'CMake executable' ${hostconfig_path} | cut -d ':' -f 2 | xargs` # Build if [[ "${option}" != "--deps-only" && "${option}" != "--test-only" ]] then + date + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ Host-config: ${hostconfig_path}" + echo "~~~~~ Build Dir: ${build_dir}" + echo "~~~~~ Project Dir: ${project_dir}" + echo "~~~~~ Install Dir: ${install_dir}" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "" echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" echo "~~~~~ Building CHAI" echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + # Map CPU core allocations + declare -A core_counts=(["lassen"]=40 ["ruby"]=28 ["poodle"]=28 ["corona"]=32 ["rzansel"]=48 ["tioga"]=32) + # If building, then delete everything first + # NOTE: 'cmake --build . -j core_counts' attempts to reduce individual build resources. + # If core_counts does not contain hostname, then will default to '-j ', which should + # use max cores. rm -rf ${build_dir} 2>/dev/null mkdir -p ${build_dir} && cd ${build_dir} - cmake \ + date + if [[ "${truehostname}" == "corona" || "${truehostname}" == "tioga" ]] + then + module unload rocm + fi + + $cmake_exe \ -C ${hostconfig_path} \ + -DCMAKE_INSTALL_PREFIX=${install_dir} \ ${project_dir} - cmake --build . -j + if ! $cmake_exe --build . -j ${core_counts[$truehostname]} + then + echo "[Error]: compilation failed, building with verbose output..." + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ Running make VERBOSE=1" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + $cmake_exe --build . --verbose -j 1 + else + # todo this should use cmake --install once we use CMake 3.15+ everywhere + #$cmake_exe --install . + make install + fi + date + + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ CHAI Built" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" fi # Test if [[ "${option}" != "--build-only" ]] && grep -q -i "ENABLE_TESTS.*ON" ${hostconfig_path} then + date echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" echo "~~~~~ Testing CHAI" echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" if [[ ! -d ${build_dir} ]] then - echo "ERROR: Build directory not found : ${build_dir}" && exit 1 + echo "[Error]: Build directory not found : ${build_dir}" && exit 1 fi cd ${build_dir} - ctest --output-on-failure -T test 2>&1 | tee tests_output.txt + date + ctest --output-on-failure --no-compress-output -T test -VV 2>&1 | tee tests_output.txt + date no_test_str="No tests were found!!!" if [[ "$(tail -n 1 tests_output.txt)" == "${no_test_str}" ]] then - echo "ERROR: No tests were found" && exit 1 + echo "[Error]: No tests were found" && exit 1 fi echo "Copying Testing xml reports for export" tree Testing - cp Testing/*/Test.xml ${project_dir} + xsltproc -o junit.xml ${project_dir}/blt/tests/ctest-to-junit.xsl Testing/*/Test.xml + mv junit.xml ${project_dir}/junit.xml if grep -q "Errors while running CTest" ./tests_output.txt then - echo "ERROR: failure(s) while running CTest" && exit 1 + echo "[Error]: failure(s) while running CTest" && exit 1 + fi + + if [[ ! -d ${install_dir} ]] + then + echo "[Error]: Install directory not found : ${install_dir}" && exit 1 fi + + cd ${install_dir}/examples/chai/using-with-cmake + mkdir build && cd build + + if ! $cmake_exe -C ../host-config.cmake ..; then + echo "[Error]: Running $cmake_exe for using-with-cmake test" && exit 1 + fi + + if ! make; then + echo "[Error]: Running make for using-with-cmake test" && exit 1 + fi + + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "~~~~~ CHAI Tests Complete" + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + date fi + +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +echo "~~~~~ CLEAN UP" +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +make clean + +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +echo "~~~~~ Build and test completed" +echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" +date diff --git a/scripts/license.txt b/scripts/license.txt index bce2cbca..5c49bc90 100644 --- a/scripts/license.txt +++ b/scripts/license.txt @@ -1,10 +1,10 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## -Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -project contributors. See the COPYRIGHT file for details. +Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +project contributors. See the CHAI LICENSE file for details. SPDX-License-Identifier: BSD-3-Clause diff --git a/scripts/make_release_tarball.sh b/scripts/make_release_tarball.sh index cce285fe..ebad11cc 100755 --- a/scripts/make_release_tarball.sh +++ b/scripts/make_release_tarball.sh @@ -1,13 +1,13 @@ #!/bin/bash ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## TAR_CMD=gtar -VERSION=2.4.0 +VERSION=2024.02.0 git archive --prefix=chai-${VERSION}/ -o chai-${VERSION}.tar HEAD 2> /dev/null diff --git a/scripts/radiuss-spack-configs b/scripts/radiuss-spack-configs index 2f156029..12fd9cfd 160000 --- a/scripts/radiuss-spack-configs +++ b/scripts/radiuss-spack-configs @@ -1 +1 @@ -Subproject commit 2f1560293e204c494fc72538161554a552a485be +Subproject commit 12fd9cfd57c11ed0c961234833a4a8a89cefd491 diff --git a/scripts/spack_packages/chai/package.py b/scripts/spack_packages/chai/package.py deleted file mode 100644 index 28442947..00000000 --- a/scripts/spack_packages/chai/package.py +++ /dev/null @@ -1,281 +0,0 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - - -from spack import * - -import socket -import os - -from os import environ as env -from os.path import join as pjoin - -import re - -def cmake_cache_entry(name, value, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE PATH "%s")\n\n' % (name,value,comment) - - -def cmake_cache_string(name, string, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE STRING "%s")\n\n' % (name,string,comment) - - -def cmake_cache_option(name, boolean_value, comment=""): - """Generate a string for a cmake configuration option""" - - value = "ON" if boolean_value else "OFF" - return 'set(%s %s CACHE BOOL "%s")\n\n' % (name,value,comment) - - -def get_spec_path(spec, package_name, path_replacements = {}, use_bin = False) : - """Extracts the prefix path for the given spack package - path_replacements is a dictionary with string replacements for the path. - """ - - if not use_bin: - path = spec[package_name].prefix - else: - path = spec[package_name].prefix.bin - - path = os.path.realpath(path) - - for key in path_replacements: - path = path.replace(key, path_replacements[key]) - - return path - - -class Chai(CMakePackage, CudaPackage): - """ - Copy-hiding array interface for data migration between memory spaces - """ - - homepage = "https://github.com/LLNL/CHAI" - git = "https://github.com/LLNL/CHAI.git" - - version('develop', branch='develop', submodules='True') - version('master', branch='main', submodules='True') - version('2.1.1', tag='v2.1.1', submodules='True') - version('2.1.0', tag='v2.1.0', submodules='True') - version('2.0.0', tag='v2.0.0', submodules='True') - version('1.2.0', tag='v1.2.0', submodules='True') - version('1.1.0', tag='v1.1.0', submodules='True') - version('1.0', tag='v1.0', submodules='True') - - variant('shared', default=False, description='Build Shared Libs') - variant('raja', default=True, description='Build plugin for RAJA') - variant('tests', default='basic', values=('none', 'basic', 'benchmarks'), - multi=False, description='Tests to run') - variant('libcpp', default=False, description='Use libc++') - - depends_on('cmake@3.8:', type='build') - depends_on('umpire@main') - depends_on('raja@main', when="+raja") - - depends_on('cmake@3.9:', type='build', when="+cuda") - depends_on('umpire+cuda', when="+cuda") - depends_on('raja+cuda', when="+raja+cuda") - depends_on('umpire+cuda+allow-untested-versions', when="+cuda+allow-untested-versions") - depends_on('raja+cuda+allow-untested-versions', when="+raja+cuda+allow-untested-versions") - depends_on('umpire+libcpp', when='+libcpp') - - phases = ['hostconfig', 'cmake', 'build', 'install'] - - def _get_sys_type(self, spec): - sys_type = str(spec.architecture) - # if on llnl systems, we can use the SYS_TYPE - if "SYS_TYPE" in env: - sys_type = env["SYS_TYPE"] - return sys_type - - def _get_host_config_path(self, spec): - var='' - if '+cuda' in spec: - var= '-'.join([var,'cuda']) - - host_config_path = "hc-%s-%s-%s%s-%s.cmake" % (socket.gethostname().rstrip('1234567890'), - self._get_sys_type(spec), - spec.compiler, - var, - spec.dag_hash()) - dest_dir = self.stage.source_path - host_config_path = os.path.abspath(pjoin(dest_dir, host_config_path)) - return host_config_path - - def hostconfig(self, spec, prefix, py_site_pkgs_dir=None): - """ - This method creates a 'host-config' file that specifies - all of the options used to configure and build CHAI. - - For more details about 'host-config' files see: - http://software.llnl.gov/conduit/building.html - - Note: - The `py_site_pkgs_dir` arg exists to allow a package that - subclasses this package provide a specific site packages - dir when calling this function. `py_site_pkgs_dir` should - be an absolute path or `None`. - - This is necessary because the spack `site_packages_dir` - var will not exist in the base class. For more details - on this issue see: https://github.com/spack/spack/issues/6261 - """ - - ####################### - # Compiler Info - ####################### - c_compiler = env["SPACK_CC"] - cpp_compiler = env["SPACK_CXX"] - - # Even though we don't have fortran code in our project we sometimes - # use the Fortran compiler to determine which libstdc++ to use - f_compiler = "" - if "SPACK_FC" in env.keys(): - # even if this is set, it may not exist - # do one more sanity check - if os.path.isfile(env["SPACK_FC"]): - f_compiler = env["SPACK_FC"] - - ####################################################################### - # By directly fetching the names of the actual compilers we appear - # to doing something evil here, but this is necessary to create a - # 'host config' file that works outside of the spack install env. - ####################################################################### - - sys_type = self._get_sys_type(spec) - - ############################################## - # Find and record what CMake is used - ############################################## - - cmake_exe = spec['cmake'].command.path - cmake_exe = os.path.realpath(cmake_exe) - - host_config_path = self._get_host_config_path(spec) - cfg = open(host_config_path, "w") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Generated host-config - Edit at own risk!\n") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Copyright (c) 2020, Lawrence Livermore National Security, LLC and\n") - cfg.write("# other CHAI Project Developers. See the top-level LICENSE file for\n") - cfg.write("# details.\n") - cfg.write("#\n") - cfg.write("# SPDX-License-Identifier: (BSD-3-Clause) \n") - cfg.write("###################\n\n".format("#" * 60)) - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# SYS_TYPE: {0}\n".format(sys_type)) - cfg.write("# Compiler Spec: {0}\n".format(spec.compiler)) - cfg.write("# CMake executable path: %s\n" % cmake_exe) - cfg.write("#------------------\n\n".format("-" * 60)) - - ####################### - # Compiler Settings - ####################### - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# Compilers\n") - cfg.write("#------------------\n\n".format("-" * 60)) - cfg.write(cmake_cache_entry("CMAKE_C_COMPILER", c_compiler)) - cfg.write(cmake_cache_entry("CMAKE_CXX_COMPILER", cpp_compiler)) - - # use global spack compiler flags - cflags = ' '.join(spec.compiler_flags['cflags']) - if cflags: - cfg.write(cmake_cache_entry("CMAKE_C_FLAGS", cflags)) - - cxxflags = ' '.join(spec.compiler_flags['cxxflags']) - if "+libcpp" in spec: - cxxflags += ' '.join([cxxflags,"-stdlib=libc++ -DGTEST_HAS_CXXABI_H_=0"]) - if cxxflags: - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS", cxxflags)) - - if ("gfortran" in f_compiler) and ("clang" in cpp_compiler): - libdir = pjoin(os.path.dirname( - os.path.dirname(f_compiler)), "lib") - flags = "" - for _libpath in [libdir, libdir + "64"]: - if os.path.exists(_libpath): - flags += " -Wl,-rpath,{0}".format(_libpath) - description = ("Adds a missing libstdc++ rpath") - if flags: - cfg.write(cmake_cache_entry("BLT_EXE_LINKER_FLAGS", flags, - description)) - - gcc_toolchain_regex = re.compile(".*gcc-toolchain.*") - gcc_name_regex = re.compile(".*gcc-name.*") - - using_toolchain = list(filter(gcc_toolchain_regex.match, spec.compiler_flags['cxxflags'])) - using_gcc_name = list(filter(gcc_name_regex.match, spec.compiler_flags['cxxflags'])) - compilers_using_toolchain = ["pgi", "xl", "icpc"] - if any(compiler in cpp_compiler for compiler in compilers_using_toolchain): - if using_toolchain or using_gcc_name: - cfg.write(cmake_cache_entry("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", - "/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3")) - - if "+cuda" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# Cuda\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_CUDA", True)) - - cudatoolkitdir = spec['cuda'].prefix - cfg.write(cmake_cache_entry("CUDA_TOOLKIT_ROOT_DIR", - cudatoolkitdir)) - cudacompiler = "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" - cfg.write(cmake_cache_entry("CMAKE_CUDA_COMPILER", - cudacompiler)) - - if not spec.satisfies('cuda_arch=none'): - cuda_arch = spec.variants['cuda_arch'].value - cuda_arch = "sm_{0}".format(cuda_arch[0]) - flag = '-arch {0}'.format(cuda_arch) - cfg.write(cmake_cache_string("CUDA_ARCH",cuda_arch)) - cfg.write(cmake_cache_string("CMAKE_CUDA_FLAGS", flag)) - - else: - cfg.write(cmake_cache_option("ENABLE_CUDA", False)) - - if "+raja" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# RAJA\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_RAJA_PLUGIN", True)) - raja_dir = spec['raja'].prefix - cfg.write(cmake_cache_entry("RAJA_DIR", raja_dir)) - else: - cfg.write(cmake_cache_option("ENABLE_RAJA_PLUGIN", False)) - - # shared vs static libs - cfg.write(cmake_cache_option("BUILD_SHARED_LIBS","+shared" in spec)) - - umpire_conf_path = spec['umpire'].prefix + "/share/umpire/cmake" - cfg.write(cmake_cache_entry("umpire_DIR",umpire_conf_path)) - - cfg.write(cmake_cache_option("ENABLE_BENCHMARKS", 'tests=benchmarks' in spec)) - cfg.write(cmake_cache_option("ENABLE_TESTS", not 'tests=none' in spec)) - - ####################### - # Close and save - ####################### - cfg.write("\n") - cfg.close() - - print("OUT: host-config file {0}".format(host_config_path)) - - def cmake_args(self): - spec = self.spec - host_config_path = self._get_host_config_path(spec) - - options = [] - options.extend(['-C', host_config_path]) - - return options diff --git a/scripts/spack_packages/raja/package.py b/scripts/spack_packages/raja/package.py deleted file mode 100644 index c7ea7117..00000000 --- a/scripts/spack_packages/raja/package.py +++ /dev/null @@ -1,336 +0,0 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - - -from spack import * - -import socket -import os - -from os import environ as env -from os.path import join as pjoin - -import re - -def cmake_cache_entry(name, value, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE PATH "%s")\n\n' % (name,value,comment) - - -def cmake_cache_string(name, string, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE STRING "%s")\n\n' % (name,string,comment) - - -def cmake_cache_option(name, boolean_value, comment=""): - """Generate a string for a cmake configuration option""" - - value = "ON" if boolean_value else "OFF" - return 'set(%s %s CACHE BOOL "%s")\n\n' % (name,value,comment) - - -def get_spec_path(spec, package_name, path_replacements = {}, use_bin = False) : - """Extracts the prefix path for the given spack package - path_replacements is a dictionary with string replacements for the path. - """ - - if not use_bin: - path = spec[package_name].prefix - else: - path = spec[package_name].prefix.bin - - path = os.path.realpath(path) - - for key in path_replacements: - path = path.replace(key, path_replacements[key]) - - return path - - -class Raja(CMakePackage, CudaPackage): - """RAJA Parallel Framework.""" - - homepage = "http://software.llnl.gov/RAJA/" - git = "https://github.com/LLNL/RAJA.git" - - version('develop', branch='develop', submodules='True') - version('main', branch='main', submodules='True') - version('0.12.1', tag='v0.12.1', submodules="True") - version('0.12.0', tag='v0.12.0', submodules="True") - version('0.11.0', tag='v0.11.0', submodules="True") - version('0.10.1', tag='v0.10.1', submodules="True") - version('0.10.0', tag='v0.10.0', submodules="True") - version('0.9.0', tag='v0.9.0', submodules="True") - version('0.8.0', tag='v0.8.0', submodules="True") - version('0.7.0', tag='v0.7.0', submodules="True") - version('0.6.0', tag='v0.6.0', submodules="True") - version('0.5.3', tag='v0.5.3', submodules="True") - version('0.5.2', tag='v0.5.2', submodules="True") - version('0.5.1', tag='v0.5.1', submodules="True") - version('0.5.0', tag='v0.5.0', submodules="True") - version('0.4.1', tag='v0.4.1', submodules="True") - version('0.4.0', tag='v0.4.0', submodules="True") - - variant('openmp', default=False, description='Build OpenMP backend') - variant('shared', default=False, description='Build Shared Libs') - variant('libcpp', default=False, description='Uses libc++ instead of libstdc++') - variant('hip', default=False, description='Build with HIP support') - variant('tests', default='none', values=('none', 'basic', 'benchmarks'), - multi=False, description='Tests to run') - - depends_on('cmake@3.8:', type='build') - depends_on('cmake@3.9:', when='+cuda', type='build') - depends_on('hip', when='+hip') - - conflicts('+openmp', when='+hip') - - phases = ['hostconfig', 'cmake', 'build', 'install'] - - def _get_sys_type(self, spec): - sys_type = str(spec.architecture) - # if on llnl systems, we can use the SYS_TYPE - if "SYS_TYPE" in env: - sys_type = env["SYS_TYPE"] - return sys_type - - def _get_host_config_path(self, spec): - var='' - if '+cuda' in spec: - var= '-'.join([var,'cuda']) - if '+libcpp' in spec: - var='-'.join([var,'libcpp']) - - host_config_path = "hc-%s-%s-%s%s-%s.cmake" % (socket.gethostname().rstrip('1234567890'), - self._get_sys_type(spec), - spec.compiler, - var, - spec.dag_hash()) - dest_dir = self.stage.source_path - host_config_path = os.path.abspath(pjoin(dest_dir, host_config_path)) - return host_config_path - - def hostconfig(self, spec, prefix, py_site_pkgs_dir=None): - """ - This method creates a 'host-config' file that specifies - all of the options used to configure and build Umpire. - - For more details about 'host-config' files see: - http://software.llnl.gov/conduit/building.html - - Note: - The `py_site_pkgs_dir` arg exists to allow a package that - subclasses this package provide a specific site packages - dir when calling this function. `py_site_pkgs_dir` should - be an absolute path or `None`. - - This is necessary because the spack `site_packages_dir` - var will not exist in the base class. For more details - on this issue see: https://github.com/spack/spack/issues/6261 - """ - - ####################### - # Compiler Info - ####################### - c_compiler = env["SPACK_CC"] - cpp_compiler = env["SPACK_CXX"] - - # Even though we don't have fortran code in our project we sometimes - # use the Fortran compiler to determine which libstdc++ to use - f_compiler = "" - if "SPACK_FC" in env.keys(): - # even if this is set, it may not exist - # do one more sanity check - if os.path.isfile(env["SPACK_FC"]): - f_compiler = env["SPACK_FC"] - - ####################################################################### - # By directly fetching the names of the actual compilers we appear - # to doing something evil here, but this is necessary to create a - # 'host config' file that works outside of the spack install env. - ####################################################################### - - sys_type = self._get_sys_type(spec) - - ############################################## - # Find and record what CMake is used - ############################################## - - cmake_exe = spec['cmake'].command.path - cmake_exe = os.path.realpath(cmake_exe) - - host_config_path = self._get_host_config_path(spec) - cfg = open(host_config_path, "w") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Generated host-config - Edit at own risk!\n") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Copyright (c) 2020, Lawrence Livermore National Security, LLC and\n") - cfg.write("# other Umpire Project Developers. See the top-level LICENSE file for\n") - cfg.write("# details.\n") - cfg.write("#\n") - cfg.write("# SPDX-License-Identifier: (BSD-3-Clause) \n") - cfg.write("###################\n\n".format("#" * 60)) - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# SYS_TYPE: {0}\n".format(sys_type)) - cfg.write("# Compiler Spec: {0}\n".format(spec.compiler)) - cfg.write("# CMake executable path: %s\n" % cmake_exe) - cfg.write("#------------------\n\n".format("-" * 60)) - - cfg.write(cmake_cache_string("CMAKE_BUILD_TYPE", spec.variants['build_type'].value)) - - ####################### - # Compiler Settings - ####################### - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# Compilers\n") - cfg.write("#------------------\n\n".format("-" * 60)) - cfg.write(cmake_cache_entry("CMAKE_C_COMPILER", c_compiler)) - cfg.write(cmake_cache_entry("CMAKE_CXX_COMPILER", cpp_compiler)) - - # use global spack compiler flags - cflags = ' '.join(spec.compiler_flags['cflags']) - if "+libcpp" in spec: - cflags += ' '.join([cflags,"-DGTEST_HAS_CXXABI_H_=0"]) - if cflags: - cfg.write(cmake_cache_entry("CMAKE_C_FLAGS", cflags)) - - cxxflags = ' '.join(spec.compiler_flags['cxxflags']) - if "+libcpp" in spec: - cxxflags += ' '.join([cxxflags,"-stdlib=libc++ -DGTEST_HAS_CXXABI_H_=0"]) - if cxxflags: - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS", cxxflags)) - - # TODO (bernede1@llnl.gov): Is this useful for RAJA? - if ("gfortran" in f_compiler) and ("clang" in cpp_compiler): - libdir = pjoin(os.path.dirname( - os.path.dirname(f_compiler)), "lib") - flags = "" - for _libpath in [libdir, libdir + "64"]: - if os.path.exists(_libpath): - flags += " -Wl,-rpath,{0}".format(_libpath) - description = ("Adds a missing libstdc++ rpath") - if flags: - cfg.write(cmake_cache_string("BLT_EXE_LINKER_FLAGS", flags, - description)) - - gcc_toolchain_regex = re.compile("--gcc-toolchain=(.*)") - gcc_name_regex = re.compile(".*gcc-name.*") - - using_toolchain = list(filter(gcc_toolchain_regex.match, spec.compiler_flags['cxxflags'])) - if(using_toolchain): - gcc_toolchain_path = gcc_toolchain_regex.match(using_toolchain[0]) - using_gcc_name = list(filter(gcc_name_regex.match, spec.compiler_flags['cxxflags'])) - compilers_using_toolchain = ["pgi", "xl", "icpc"] - if any(compiler in cpp_compiler for compiler in compilers_using_toolchain): - if using_toolchain or using_gcc_name: - cfg.write(cmake_cache_entry("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", - "/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3")) - - if "+cuda" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# Cuda\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_CUDA", True)) - - cudatoolkitdir = spec['cuda'].prefix - cfg.write(cmake_cache_entry("CUDA_TOOLKIT_ROOT_DIR", - cudatoolkitdir)) - cudacompiler = "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" - cfg.write(cmake_cache_entry("CMAKE_CUDA_COMPILER", - cudacompiler)) - - cuda_release_flags = "-O3 -Xcompiler -Ofast -Xcompiler -finline-functions -Xcompiler -finline-limit=20000" - cuda_reldebinf_flags = "-O3 -g -Xcompiler -Ofast -Xcompiler -finline-functions -Xcompiler -finline-limit=20000" - cuda_debug_flags = "-O0 -g -Xcompiler -O0 -Xcompiler -finline-functions -Xcompiler -finline-limit=20000" - - cfg.write(cmake_cache_string("CMAKE_CUDA_FLAGS_RELEASE", cuda_release_flags)) - cfg.write(cmake_cache_string("CMAKE_CUDA_FLAGS_RELWITHDEBINFO", cuda_reldebinf_flags)) - cfg.write(cmake_cache_string("CMAKE_CUDA_FLAGS_DEBUG", cuda_debug_flags)) - - if not spec.satisfies('cuda_arch=none'): - cuda_arch = spec.variants['cuda_arch'].value - cfg.write(cmake_cache_string("CUDA_ARCH", 'sm_{0}'.format(cuda_arch[0]))) - - else: - cfg.write(cmake_cache_option("ENABLE_CUDA", False)) - - if "+hip" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# HIP\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_HIP", True)) - - hip_root = spec['hip'].prefix - rocm_root = hip_root + "/.." - cfg.write(cmake_cache_entry("HIP_ROOT_DIR", - hip_root)) - cfg.write(cmake_cache_entry("HIP_CLANG_PATH", - rocm_root + '/llvm/bin')) - cfg.write(cmake_cache_entry("HIP_HIPCC_FLAGS", - '--amdgpu-target=gfx906')) - cfg.write(cmake_cache_entry("HIP_RUNTIME_INCLUDE_DIRS", - "{0}/include;{0}/../hsa/include".format(hip_root))) - if ('%gcc' in spec) or (using_toolchain): - if ('%gcc' in spec): - gcc_bin = os.path.dirname(self.compiler.cxx) - gcc_prefix = join_path(gcc_bin, '..') - else: - gcc_prefix = gcc_toolchain_path.group(1) - cfg.write(cmake_cache_entry("HIP_CLANG_FLAGS", - "--gcc-toolchain={0}".format(gcc_prefix))) - cfg.write(cmake_cache_entry("CMAKE_EXE_LINKER_FLAGS", - "-Wl,-rpath {}/lib64".format(gcc_prefix))) - - else: - cfg.write(cmake_cache_option("ENABLE_HIP", False)) - - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# Other\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_string("RAJA_RANGE_ALIGN", "4")) - cfg.write(cmake_cache_string("RAJA_RANGE_MIN_LENGTH", "32")) - cfg.write(cmake_cache_string("RAJA_DATA_ALIGN", "64")) - - cfg.write(cmake_cache_option("RAJA_HOST_CONFIG_LOADED", True)) - - # shared vs static libs - cfg.write(cmake_cache_option("BUILD_SHARED_LIBS","+shared" in spec)) - cfg.write(cmake_cache_option("ENABLE_OPENMP","+openmp" in spec)) - - # Note 1: Work around spack adding -march=ppc64le to SPACK_TARGET_ARGS - # which is used by the spack compiler wrapper. This can go away when - # BLT removes -Werror from GTest flags - # Note 2: Tests are either built if variant is set, or if run-tests - # option is passed. - if self.spec.satisfies('%clang target=ppc64le:'): - cfg.write(cmake_cache_option("ENABLE_TESTS",False)) - if 'tests=benchmarks' in spec or not 'tests=none' in spec: - print("MSG: no testing supported on %clang target=ppc64le:") - else: - cfg.write(cmake_cache_option("ENABLE_BENCHMARKS", 'tests=benchmarks' in spec)) - cfg.write(cmake_cache_option("ENABLE_TESTS", not 'tests=none' in spec or self.run_tests)) - - ####################### - # Close and save - ####################### - cfg.write("\n") - cfg.close() - - print("OUT: host-config file {0}".format(host_config_path)) - - def cmake_args(self): - spec = self.spec - host_config_path = self._get_host_config_path(spec) - - options = [] - options.extend(['-C', host_config_path]) - - return options diff --git a/scripts/spack_packages/umpire/camp_target_umpire_3.0.0.patch b/scripts/spack_packages/umpire/camp_target_umpire_3.0.0.patch deleted file mode 100644 index 2deb565c..00000000 --- a/scripts/spack_packages/umpire/camp_target_umpire_3.0.0.patch +++ /dev/null @@ -1,18 +0,0 @@ -diff --git a/umpire-config.cmake.in b/umpire-config.cmake.in -index a98ad5fe..4e54e173 100644 ---- a/umpire-config.cmake.in -+++ b/umpire-config.cmake.in -@@ -7,6 +7,13 @@ - get_filename_component(UMPIRE_CMAKE_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH) - set(UMPIRE_INCLUDE_DIRS "@CMAKE_INSTALL_PREFIX@/include") - -+if (NOT TARGET camp) -+ if (NOT DEFINED camp_DIR) -+ set(camp_DIR @CMAKE_INSTALL_PREFIX@/lib/cmake/camp) -+ endif () -+ find_package(camp REQUIRED) -+endif () -+ - set(Umpire_VERSION_MAJOR @Umpire_VERSION_MAJOR@) - set(Umpire_VERSION_MINOR @Umpire_VERSION_MINOR@) - set(Umpire_VERSION_PATCH @Umpire_VERSION_PATCH@) diff --git a/scripts/spack_packages/umpire/package.py b/scripts/spack_packages/umpire/package.py deleted file mode 100644 index 8bc83634..00000000 --- a/scripts/spack_packages/umpire/package.py +++ /dev/null @@ -1,396 +0,0 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - - -from spack import * - -import socket -import os - -from os import environ as env -from os.path import join as pjoin - -import re - -def cmake_cache_entry(name, value, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE PATH "%s")\n\n' % (name,value,comment) - - -def cmake_cache_string(name, string, comment=""): - """Generate a string for a cmake cache variable""" - - return 'set(%s "%s" CACHE STRING "%s")\n\n' % (name,string,comment) - - -def cmake_cache_option(name, boolean_value, comment=""): - """Generate a string for a cmake configuration option""" - - value = "ON" if boolean_value else "OFF" - return 'set(%s %s CACHE BOOL "%s")\n\n' % (name,value,comment) - - -def get_spec_path(spec, package_name, path_replacements = {}, use_bin = False) : - """Extracts the prefix path for the given spack package - path_replacements is a dictionary with string replacements for the path. - """ - - if not use_bin: - path = spec[package_name].prefix - else: - path = spec[package_name].prefix.bin - - path = os.path.realpath(path) - - for key in path_replacements: - path = path.replace(key, path_replacements[key]) - - return path - -class Umpire(CMakePackage, CudaPackage): - """An application-focused API for memory management on NUMA & GPU - architectures""" - - homepage = 'https://github.com/LLNL/Umpire' - git = 'https://github.com/LLNL/Umpire.git' - - version('develop', branch='develop', submodules='True') - version('main', branch='main', submodules='True') - version('4.1.2', tag='v4.1.2', submodules='True') - version('3.0.0', tag='v3.0.0', submodules='True') - version('2.1.0', tag='v2.1.0', submodules='True') - version('2.0.0', tag='v2.0.0', submodules='True') - version('1.1.0', tag='v1.1.0', submodules='True') - version('1.0.1', tag='v1.0.1', submodules='True') - version('1.0.0', tag='v1.0.0', submodules='True') - version('0.3.5', tag='v0.3.5', submodules='True') - version('0.3.4', tag='v0.3.4', submodules='True') - version('0.3.3', tag='v0.3.3', submodules='True') - version('0.3.2', tag='v0.3.2', submodules='True') - version('0.3.1', tag='v0.3.1', submodules='True') - version('0.3.0', tag='v0.3.0', submodules='True') - version('0.2.4', tag='v0.2.4', submodules='True') - version('0.2.3', tag='v0.2.3', submodules='True') - version('0.2.2', tag='v0.2.2', submodules='True') - version('0.2.1', tag='v0.2.1', submodules='True') - version('0.2.0', tag='v0.2.0', submodules='True') - version('0.1.4', tag='v0.1.4', submodules='True') - version('0.1.3', tag='v0.1.3', submodules='True') - - patch('camp_target_umpire_3.0.0.patch', when='@3.0.0') - - variant('fortran', default=False, description='Build C/Fortran API') - variant('c', default=False, description='Build C API') - variant('mpi', default=False, description='Enable MPI support') - variant('numa', default=False, description='Enable NUMA support') - variant('shared', default=False, description='Enable Shared libs') - variant('openmp', default=False, description='Build with OpenMP support') - variant('openmp_target', default=False, description='Build with OpenMP 4.5 support') - variant('deviceconst', default=False, - description='Enables support for constant device memory') - variant('tests', default='none', values=('none', 'basic', 'benchmarks'), - multi=False, description='Tests to run') - - variant('libcpp', default=False, description='Uses libc++ instead of libstdc++') - variant('hip', default=False, description='Build with HIP support') - variant('tools', default=False, description='Enable tools') - variant('dev_benchmarks', default=False, description='Enable Developer Benchmarks') - variant('werror', default=False, description='Enable warnings as errors') - variant('asan', default=False, description='Enable ASAN') - variant('sanitizer_tests', default=False, description='Enable address sanitizer tests') - - depends_on('cmake@3.8:', type='build') - depends_on('cmake@3.9:', when='+cuda', type='build') - depends_on('mpi', when='+mpi') - depends_on('hip', when='+hip') - - conflicts('+numa', when='@:0.3.2') - conflicts('~c', when='+fortran', msg='Fortran API requires C API') - conflicts('~openmp', when='+openmp_target', msg='OpenMP target requires OpenMP') - conflicts('+cuda', when='+hip') - conflicts('+openmp', when='+hip') - conflicts('+openmp_target', when='+hip') - conflicts('+deviceconst', when='~hip~cuda') - conflicts('+sanitizer_tests', when='~asan') - - phases = ['hostconfig', 'cmake', 'build', 'install'] - - def _get_sys_type(self, spec): - sys_type = str(spec.architecture) - # if on llnl systems, we can use the SYS_TYPE - if "SYS_TYPE" in env: - sys_type = env["SYS_TYPE"] - return sys_type - - def _get_host_config_path(self, spec): - var='' - if '+cuda' in spec: - var= '-'.join([var,'cuda']) - if '+libcpp' in spec: - var='-'.join([var,'libcpp']) - - host_config_path = "hc-%s-%s-%s%s-%s.cmake" % (socket.gethostname().rstrip('1234567890'), - self._get_sys_type(spec), - spec.compiler, - var, - spec.dag_hash()) - dest_dir = self.stage.source_path - host_config_path = os.path.abspath(pjoin(dest_dir, host_config_path)) - return host_config_path - - def hostconfig(self, spec, prefix, py_site_pkgs_dir=None): - """ - This method creates a 'host-config' file that specifies - all of the options used to configure and build Umpire. - - For more details about 'host-config' files see: - http://software.llnl.gov/conduit/building.html - - Note: - The `py_site_pkgs_dir` arg exists to allow a package that - subclasses this package provide a specific site packages - dir when calling this function. `py_site_pkgs_dir` should - be an absolute path or `None`. - - This is necessary because the spack `site_packages_dir` - var will not exist in the base class. For more details - on this issue see: https://github.com/spack/spack/issues/6261 - """ - - ####################### - # Compiler Info - ####################### - c_compiler = env["SPACK_CC"] - cpp_compiler = env["SPACK_CXX"] - - # Even though we don't have fortran code in our project we sometimes - # use the Fortran compiler to determine which libstdc++ to use - f_compiler = "" - if "SPACK_FC" in env.keys(): - # even if this is set, it may not exist - # do one more sanity check - if os.path.isfile(env["SPACK_FC"]): - f_compiler = env["SPACK_FC"] - - - ####################################################################### - # By directly fetching the names of the actual compilers we appear - # to doing something evil here, but this is necessary to create a - # 'host config' file that works outside of the spack install env. - ####################################################################### - - sys_type = self._get_sys_type(spec) - - ############################################## - # Find and record what CMake is used - ############################################## - - cmake_exe = spec['cmake'].command.path - cmake_exe = os.path.realpath(cmake_exe) - - host_config_path = self._get_host_config_path(spec) - cfg = open(host_config_path, "w") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Generated host-config - Edit at own risk!\n") - cfg.write("###################\n".format("#" * 60)) - cfg.write("# Copyright (c) 2020, Lawrence Livermore National Security, LLC and\n") - cfg.write("# other Umpire Project Developers. See the top-level LICENSE file for\n") - cfg.write("# details.\n") - cfg.write("#\n") - cfg.write("# SPDX-License-Identifier: (BSD-3-Clause) \n") - cfg.write("###################\n\n".format("#" * 60)) - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# SYS_TYPE: {0}\n".format(sys_type)) - cfg.write("# Compiler Spec: {0}\n".format(spec.compiler)) - cfg.write("# CMake executable path: %s\n" % cmake_exe) - cfg.write("#------------------\n\n".format("-" * 60)) - - cfg.write(cmake_cache_string("CMAKE_BUILD_TYPE", spec.variants['build_type'].value)) - - ####################### - # Compiler Settings - ####################### - - cfg.write("#------------------\n".format("-" * 60)) - cfg.write("# Compilers\n") - cfg.write("#------------------\n\n".format("-" * 60)) - cfg.write(cmake_cache_entry("CMAKE_C_COMPILER", c_compiler)) - cfg.write(cmake_cache_entry("CMAKE_CXX_COMPILER", cpp_compiler)) - if '+fortran' in spec: - cfg.write(cmake_cache_entry("CMAKE_Fortran_COMPILER", f_compiler)) - - # use global spack compiler flags - cflags = ' '.join(spec.compiler_flags['cflags']) - if "+libcpp" in spec: - cflags += ' '.join([cflags,"-DGTEST_HAS_CXXABI_H_=0"]) - if cflags: - cfg.write(cmake_cache_entry("CMAKE_C_FLAGS", cflags)) - - cxxflags = ' '.join(spec.compiler_flags['cxxflags']) - if "+libcpp" in spec: - cxxflags += ' '.join([cxxflags,"-stdlib=libc++ -DGTEST_HAS_CXXABI_H_=0"]) - if cxxflags: - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS", cxxflags)) - - fflags = ' '.join(spec.compiler_flags['fflags']) - cfg.write(cmake_cache_entry("CMAKE_Fortran_FLAGS", fflags)) - - fortran_compilers = ["gfortran", "xlf"] - if any(compiler in f_compiler for compiler in fortran_compilers) and ("clang" in cpp_compiler): - cfg.write(cmake_cache_entry("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", - "/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3")) - - libdir = pjoin(os.path.dirname( - os.path.dirname(f_compiler)), "lib") - flags = "" - for _libpath in [libdir, libdir + "64"]: - if os.path.exists(_libpath): - flags += " -Wl,-rpath,{0}".format(_libpath) - description = ("Adds a missing libstdc++ rpath") - if flags: - cfg.write(cmake_cache_entry("BLT_EXE_LINKER_FLAGS", flags, - description)) - - - gcc_toolchain_regex = re.compile(".*gcc-toolchain.*") - gcc_name_regex = re.compile(".*gcc-name.*") - - using_toolchain = list(filter(gcc_toolchain_regex.match, spec.compiler_flags['cxxflags'])) - using_gcc_name = list(filter(gcc_name_regex.match, spec.compiler_flags['cxxflags'])) - compilers_using_toolchain = ["pgi", "xl", "icpc"] - if any(compiler in cpp_compiler for compiler in compilers_using_toolchain): - if using_toolchain or using_gcc_name: - cfg.write(cmake_cache_entry("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", - "/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3")) - - if "toss_3_x86_64_ib" in sys_type: - release_flags = "-O3" - reldebinf_flags = "-O3 -g" - debug_flags = "-O0 -g" - - if "intel" in str(spec.compiler): - release_flags = ' '.join([release_flags,'-finline-functions -axCORE-AVX2 -diag-disable cpu-dispatch']) - reldebinf_flags = ' '.join([reldebinf_flags,'-finline-functions -axCORE-AVX2 -diag-disable cpu-dispatch']) - - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS_RELEASE", release_flags)) - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS_RELWITHDEBINFO", reldebinf_flags)) - cfg.write(cmake_cache_entry("CMAKE_CXX_FLAGS_DEBUG", debug_flags)) - - #Configuration to enable developer benchmarks (i.e. No-Op) - if "+dev_benchmarks" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# Developer Benchmarks\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_DEVELOPER_BENCHMARKS", True)) - cfg.write(cmake_cache_option("ENABLE_BENCHMARKS", True)) #Enable BLT GoogleBenchmark support - else: - cfg.write(cmake_cache_option("ENABLE_DEVELOPER_BENCHMARKS", False)) - cfg.write(cmake_cache_option("ENABLE_BENCHMARKS", False)) - - - if "+cuda" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# Cuda\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_CUDA", True)) - - cudatoolkitdir = spec['cuda'].prefix - cfg.write(cmake_cache_entry("CUDA_TOOLKIT_ROOT_DIR", - cudatoolkitdir)) - cudacompiler = "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" - cfg.write(cmake_cache_entry("CMAKE_CUDA_COMPILER", - cudacompiler)) - - cuda_flags = [] - - if not spec.satisfies('cuda_arch=none'): - cuda_arch = spec.variants['cuda_arch'].value - cuda_flags.append('-arch sm_{0}'.format(cuda_arch[0])) - - if '+deviceconst' in spec: - cfg.write(cmake_cache_option("ENABLE_DEVICE_CONST", True)) - - if using_toolchain: - cuda_flags.append("-Xcompiler {}".format(using_toolchain[0])) - - cfg.write(cmake_cache_string("CMAKE_CUDA_FLAGS", ' '.join(cuda_flags))) - - else: - cfg.write(cmake_cache_option("ENABLE_CUDA", False)) - - if "+hip" in spec: - cfg.write("#------------------{0}\n".format("-" * 60)) - cfg.write("# HIP\n") - cfg.write("#------------------{0}\n\n".format("-" * 60)) - - cfg.write(cmake_cache_option("ENABLE_HIP", True)) - -# -DHIP_ROOT_DIR=/opt/rocm-3.6.0/hip -DHIP_CLANG_PATH=/opt/rocm-3.6.0/llvm/bin - - hip_root = spec['hip'].prefix - rocm_root = hip_root + "/.." - cfg.write(cmake_cache_entry("HIP_ROOT_DIR", - hip_root)) - cfg.write(cmake_cache_entry("HIP_CLANG_PATH", - rocm_root + '/llvm/bin')) - cfg.write(cmake_cache_entry("HIP_HIPCC_FLAGS", - '--amdgpu-target=gfx906')) - cfg.write(cmake_cache_entry("HIP_RUNTIME_INCLUDE_DIRS", - "{0}/include;{0}/../hsa/include".format(hip_root))) - if '%gcc' in spec: - gcc_bin = os.path.dirname(self.compiler.cxx) - gcc_prefix = join_path(gcc_bin, '..') - cfg.write(cmake_cache_entry("HIP_CLANG_FLAGS", - "--gcc-toolchain={0}".format(gcc_prefix))) - cfg.write(cmake_cache_entry("CMAKE_EXE_LINKER_FLAGS", - "-Wl,-rpath {}/lib64".format(gcc_prefix))) - - if '+deviceconst' in spec: - cfg.write(cmake_cache_option("ENABLE_DEVICE_CONST", True)) - - else: - cfg.write(cmake_cache_option("ENABLE_HIP", False)) - - cfg.write(cmake_cache_option("ENABLE_C", '+c' in spec)) - cfg.write(cmake_cache_option("ENABLE_FORTRAN", '+fortran' in spec)) - - if "+mpi" in spec: - cfg.write(cmake_cache_option("ENABLE_MPI", '+mpi' in spec)) - cfg.write(cmake_cache_entry("MPI_CXX_COMPILER", spec['mpi'].mpicxx)) - - cfg.write(cmake_cache_option("ENABLE_NUMA", '+numa' in spec)) - cfg.write(cmake_cache_option("ENABLE_OPENMP", '+openmp' in spec)) - if "+openmp_target" in spec: - cfg.write(cmake_cache_option("ENABLE_OPENMP_TARGET", True)) - if ('%xl' in spec): - cfg.write(cmake_cache_entry("OpenMP_CXX_FLAGS", "-qsmp;-qoffload")) - - cfg.write(cmake_cache_option("ENABLE_BENCHMARKS", 'tests=benchmarks' in spec)) - cfg.write(cmake_cache_option("ENABLE_TESTS", not 'tests=none' in spec)) - cfg.write(cmake_cache_option("ENABLE_TOOLS", '+tools' in spec)) - cfg.write(cmake_cache_option("ENABLE_WARNINGS_AS_ERRORS", '+werror' in spec)) - cfg.write(cmake_cache_option("ENABLE_ASAN", '+asan' in spec)) - cfg.write(cmake_cache_option("ENABLE_SANITIZER_TESTS", '+sanitizer_tests' in spec)) - - ####################### - # Close and save - ####################### - cfg.write("\n") - cfg.close() - - print("OUT: host-config file {0}".format(host_config_path)) - - def cmake_args(self): - spec = self.spec - host_config_path = self._get_host_config_path(spec) - - options = [] - options.extend(['-C', host_config_path]) - - return options diff --git a/scripts/travis/build_and_test.sh b/scripts/travis/build_and_test.sh index c96efaa5..eb9879fd 100755 --- a/scripts/travis/build_and_test.sh +++ b/scripts/travis/build_and_test.sh @@ -1,7 +1,7 @@ #!/bin/bash ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/scripts/travis/install_llvm.sh b/scripts/travis/install_llvm.sh index 499f3a57..79db6804 100755 --- a/scripts/travis/install_llvm.sh +++ b/scripts/travis/install_llvm.sh @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/scripts/uberenv b/scripts/uberenv index 105e384f..0d00dc8e 160000 --- a/scripts/uberenv +++ b/scripts/uberenv @@ -1 +1 @@ -Subproject commit 105e384f585e2391c42b2def93124a6580319c1c +Subproject commit 0d00dc8e19a889ba07ae433590b87533c4b5b3da diff --git a/scripts/update-copyright-year.sh b/scripts/update-copyright-year.sh deleted file mode 100755 index d4cfd693..00000000 --- a/scripts/update-copyright-year.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env zsh -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: BSD-3-Clause -############################################################################## -# This is used for the ~*tpl* line to ignore files in bundled tpls -setopt extended_glob - -autoload colors - -RED="\033[1;31m" -GREEN="\033[1;32m" -NOCOLOR="\033[0m" - -files_no_license=$(grep -l '2016-19,' \ - benchmarks/**/*(^/) \ - cmake/**/*(^/) \ - docs/**/*~*rst(^/)\ - examples/**/*(^/) \ - scripts/**/*~*copyright*(^/) \ - src/**/*~*tpl*(^/) \ - tests/**/*(^/) \ - CMakeLists.txt) - -if [ $files_no_license ]; then - print "${RED} [!] Some files need copyright year updating: ${NOCOLOR}" - echo "${files_no_license}" - - echo ${files_no_license} | xargs sed -i '' 's/2016-19,/2016-20,/' - - print "${GREEN} [Ok] Copyright years updated." - - exit 0 -else - print "${GREEN} [Ok] All files have required license info." - exit 0 -fi diff --git a/scripts/update-copyright.sh b/scripts/update-copyright.sh new file mode 100755 index 00000000..870b6d66 --- /dev/null +++ b/scripts/update-copyright.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash + +############################################################################## +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## + +#============================================================================= +# Change the copyright date in all files that contain the text +# "the CHAI LICENSE file", which is part of the copyright statement at the +# top of each CHAI file. We use this to distinguish CHAI files from ones +# that we do not own (e.g., other repos included as submodules), which we +# do not want to modify. Note that this file and *.git files are omitted +# as well. +# +# IMPORTANT: Since this file is not modified (it is running the shell +# script commands), you must EDIT THE COPYRIGHT DATES IN THE HEADER ABOVE +# MANUALLY. +# +# Edit the 'grep' command below to change the set of files that will be +# modified. +# +# Change the 'sed' command below to change the content that is changed +# in each file and what it is changed to. Typically, this would only be +# modifying the end year in the copyright date. +# +#============================================================================= +# +# If you need to modify this script, you may want to run each of these +# commands individually from the command line to make sure things are doing +# what you think they should be doing. This is why they are separated into +# steps here. +# +#============================================================================= + +#============================================================================= +# First find all the files we want to modify +#============================================================================= +grep -rl "the CHAI LICENSE file" . --exclude-dir=.git --exclude-dir=blt --exclude-dir=umpire --exclude-dir=raja --exclude-dir=radiuss-spack-configs --exclude-dir=uberenv --exclude=update-copyright.sh > files2change + +#============================================================================= +# Replace the old copyright dates with new dates +#============================================================================= +for i in `cat files2change` +do + echo $i + cp $i $i.sed.bak + sed "s/Copyright (c) \([0-9]\{4\}\)-[0-9]\{2\},/Copyright (c) \1-24,/" $i.sed.bak > $i +done + +echo LICENSE +cp LICENSE LICENSE.sed.bak +sed "s/Copyright (c) \([0-9]\{4\}\)-[0-9]\{4\}/Copyright (c) \1-2024/" LICENSE.sed.bak > LICENSE + +for i in README.md CONTRIBUTING.md +do + echo $i + cp $i $i.sed.bak + sed "s/\([0-9]\{4\}\)-[0-9]\{2\}/\1-24/" $i.sed.bak > $i +done + +#============================================================================= +# Remove temporary files created in the process +#============================================================================= +find . -name \*.sed.bak -exec rm {} \; +rm files2change diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 8536f824..3691d2cd 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -1,6 +1,6 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## diff --git a/src/chai/ArrayManager.cpp b/src/chai/ArrayManager.cpp index 8a07c7cc..bc0fc20f 100644 --- a/src/chai/ArrayManager.cpp +++ b/src/chai/ArrayManager.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -9,13 +9,17 @@ #include "chai/config.hpp" #if defined(CHAI_ENABLE_CUDA) +#if !defined(CHAI_THIN_GPU_ALLOCATE) #include "cuda_runtime_api.h" #endif +#endif #include "umpire/ResourceManager.hpp" namespace chai { +thread_local ExecutionSpace ArrayManager::m_current_execution_space; +thread_local bool ArrayManager::m_synced_since_last_kernel = false; PointerRecord ArrayManager::s_null_record = PointerRecord(); @@ -170,6 +174,12 @@ void ArrayManager::setExecutionSpace(ExecutionSpace space) m_synced_since_last_kernel = false; } +#if defined(CHAI_THIN_GPU_ALLOCATE) + if (chai::CPU == space) { + syncIfNeeded(); + } +#endif + m_current_execution_space = space; } @@ -224,6 +234,34 @@ void ArrayManager::resetTouch(PointerRecord* pointer_record) } } + +/* Not all GPU platform runtimes (notably HIP), will give you asynchronous copies to the device by default, so we leverage + * umpire's API for asynchronous copies using camp resources in this method, based off of the CHAI destination space + * */ +static void copy(void * dst_pointer, void * src_pointer, umpire::ResourceManager & manager, ExecutionSpace dst_space, ExecutionSpace src_space) { + +#ifdef CHAI_ENABLE_CUDA + camp::resources::Resource device_resource(camp::resources::Cuda::get_default()); +#elif defined(CHAI_ENABLE_HIP) + camp::resources::Resource device_resource(camp::resources::Hip::get_default()); +#else + camp::resources::Resource device_resource(camp::resources::Host::get_default()); +#endif + + camp::resources::Resource host_resource(camp::resources::Host::get_default()); + if (dst_space == GPU || src_space == GPU) { + // Do the copy using the device resource + manager.copy(dst_pointer, src_pointer, device_resource); + } else { + // Do the copy using the host resource + manager.copy(dst_pointer, src_pointer, host_resource); + } + // Ensure device to host copies are synchronous + if (dst_space == CPU && src_space == GPU) { + device_resource.wait(); + } +} + void ArrayManager::move(PointerRecord* record, ExecutionSpace space) { if (space == NONE) { @@ -251,7 +289,9 @@ void ArrayManager::move(PointerRecord* record, ExecutionSpace space) } #endif - void* src_pointer = record->m_pointers[record->m_last_space]; + ExecutionSpace prev_space = record->m_last_space; + + void* src_pointer = record->m_pointers[prev_space]; void* dst_pointer = record->m_pointers[space]; if (!dst_pointer) { @@ -265,7 +305,7 @@ void ArrayManager::move(PointerRecord* record, ExecutionSpace space) } else if (dst_pointer != src_pointer) { // Exclude the copy if src and dst are the same (can happen for PINNED memory) { - m_resource_manager.copy(dst_pointer, src_pointer); + chai::copy(dst_pointer, src_pointer, m_resource_manager, space, prev_space); } callback(record, ACTION_MOVE, space); @@ -283,7 +323,6 @@ void ArrayManager::allocate( pointer_record->m_pointers[space] = alloc.allocate(size); callback(pointer_record, ACTION_ALLOC, space); - registerPointer(pointer_record, space); CHAI_LOG(Debug, "Allocated array at: " << pointer_record->m_pointers[space]); @@ -447,32 +486,32 @@ PointerRecord* ArrayManager::makeManaged(void* pointer, PointerRecord* ArrayManager::deepCopyRecord(PointerRecord const* record) { - PointerRecord* copy = new PointerRecord{}; + PointerRecord* new_record = new PointerRecord{}; const size_t size = record->m_size; - copy->m_size = size; - copy->m_user_callback = [] (const PointerRecord*, Action, ExecutionSpace) {}; + new_record->m_size = size; + new_record->m_user_callback = [] (const PointerRecord*, Action, ExecutionSpace) {}; const ExecutionSpace last_space = record->m_last_space; - copy->m_last_space = last_space; + new_record->m_last_space = last_space; for (int space = CPU; space < NUM_EXECUTION_SPACES; ++space) { - copy->m_allocators[space] = record->m_allocators[space]; + new_record->m_allocators[space] = record->m_allocators[space]; } - allocate(copy, last_space); + allocate(new_record, last_space); for (int space = CPU; space < NUM_EXECUTION_SPACES; ++space) { - copy->m_owned[space] = true; - copy->m_touched[space] = false; + new_record->m_owned[space] = true; + new_record->m_touched[space] = false; } - copy->m_touched[last_space] = true; + new_record->m_touched[last_space] = true; - void* dst_pointer = copy->m_pointers[last_space]; + void* dst_pointer = new_record->m_pointers[last_space]; void* src_pointer = record->m_pointers[last_space]; - m_resource_manager.copy(dst_pointer, src_pointer); + chai::copy(dst_pointer, src_pointer, m_resource_manager, last_space, last_space); - return copy; + return new_record; } std::unordered_map @@ -547,21 +586,23 @@ void ArrayManager::evict(ExecutionSpace space, ExecutionSpace destinationSpace) // Now move and evict std::vector pointersToEvict; - std::lock_guard lock(m_mutex); - for (const auto& entry : m_pointer_map) { - // Get the pointer record - auto record = *entry.second; - - // Move the data and register the touches - move(record, destinationSpace); - registerTouch(record, destinationSpace); - - // If the destinationSpace is ever allowed to be NONE, then we will need to - // update the touch in the eviction space and make sure the last space is not - // the eviction space. - - // Mark record for eviction later in this routine - pointersToEvict.push_back(record); + { + std::lock_guard lock(m_mutex); + for (const auto& entry : m_pointer_map) { + // Get the pointer record + auto record = *entry.second; + + // Move the data and register the touches + move(record, destinationSpace); + registerTouch(record, destinationSpace); + + // If the destinationSpace is ever allowed to be NONE, then we will need to + // update the touch in the eviction space and make sure the last space is not + // the eviction space. + + // Mark record for eviction later in this routine + pointersToEvict.push_back(record); + } } // This must be done in a second pass because free erases from m_pointer_map, diff --git a/src/chai/ArrayManager.hpp b/src/chai/ArrayManager.hpp index dcddd99a..a1c880fb 100644 --- a/src/chai/ArrayManager.hpp +++ b/src/chai/ArrayManager.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -396,21 +396,6 @@ class ArrayManager */ void disableCallbacks() { m_callbacks_active = false; } - /*! - * \brief Turn on device synchronization after every kernel. - */ - void enableDeviceSynchronize() { m_device_synchronize = true; } - - /*! - * \brief Turn off device synchronization after every kernel. - */ - void disableDeviceSynchronize() { m_device_synchronize = false; } - - /*! - * \brief Turn on device synchronization after every kernel. - */ - bool deviceSynchronize() { return m_device_synchronize; } - /*! * \brief synchronize the device if there hasn't been a synchronize since the last kernel */ @@ -487,7 +472,7 @@ class ArrayManager /*! * Current execution space. */ - ExecutionSpace m_current_execution_space; + static thread_local ExecutionSpace m_current_execution_space; /** * Default space for new allocations. @@ -525,16 +510,11 @@ class ArrayManager */ bool m_callbacks_active; - /*! - * Whether or not to synchronize on device after every CHAI kernel. - */ - bool m_device_synchronize = false; - /*! * Whether or not a synchronize has been performed since the launch of the last * GPU context */ - bool m_synced_since_last_kernel = false; + static thread_local bool m_synced_since_last_kernel; #if defined(CHAI_ENABLE_GPU_SIMULATION_MODE) /*! diff --git a/src/chai/ArrayManager.inl b/src/chai/ArrayManager.inl index 06a5253c..e2aa34db 100644 --- a/src/chai/ArrayManager.inl +++ b/src/chai/ArrayManager.inl @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -17,8 +17,10 @@ #include "umpire/ResourceManager.hpp" #if defined(CHAI_ENABLE_UM) +#if !defined(CHAI_THIN_GPU_ALLOCATE) #include #endif +#endif namespace chai { diff --git a/src/chai/CMakeLists.txt b/src/chai/CMakeLists.txt index 4ce3c41f..2285c544 100644 --- a/src/chai/CMakeLists.txt +++ b/src/chai/CMakeLists.txt @@ -1,18 +1,11 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## -set(CHAI_ENABLE_PICK ${ENABLE_PICK}) -set(CHAI_ENABLE_CUDA ${ENABLE_CUDA}) -set(CHAI_ENABLE_HIP ${ENABLE_HIP}) -set(CHAI_ENABLE_IMPLICIT_CONVERSIONS ${ENABLE_IMPLICIT_CONVERSIONS}) -set(CHAI_DISABLE_RM ${DISABLE_RM}) -set(CHAI_ENABLE_UM ${ENABLE_UM}) -set(CHAI_ENABLE_RAJA_PLUGIN ${ENABLE_RAJA_PLUGIN}) -set(CHAI_ENABLE_GPU_SIMULATION_MODE ${ENABLE_GPU_SIMULATION_MODE}) -set(CHAI_ENABLE_PINNED ${ENABLE_PINNED}) + +include(CMakePackageConfigHelpers) configure_file( ${PROJECT_SOURCE_DIR}/src/chai/config.hpp.in @@ -29,7 +22,7 @@ set (chai_headers PointerRecord.hpp Types.hpp) -if(DISABLE_RM) +if(CHAI_DISABLE_RM) set(chai_headers ${chai_headers} ManagedArray_thin.inl) @@ -41,18 +34,18 @@ set (chai_sources set (chai_depends umpire) -if (ENABLE_CUDA) +if (CHAI_ENABLE_CUDA) set (chai_depends ${chai_depends} cuda_runtime) endif () -if (ENABLE_HIP) +if (CHAI_ENABLE_HIP) set (chai_depends ${chai_depends} - hip_runtime) + blt::hip_runtime) endif () -if (ENABLE_RAJA_PLUGIN) +if (CHAI_ENABLE_RAJA_PLUGIN) set (chai_headers ${chai_headers} pluginLinker.hpp @@ -67,7 +60,7 @@ if (ENABLE_RAJA_PLUGIN) ${chai_depends} RAJA) - if (ENABLE_CUDA) + if (CHAI_ENABLE_CUDA) set (chai_depends ${chai_depends} cuda) @@ -91,13 +84,22 @@ target_include_directories( $ $) -configure_file( - ${PROJECT_SOURCE_DIR}/src/chai/chai-config.cmake.in - ${PROJECT_BINARY_DIR}/share/chai/cmake/chai-config.cmake) +configure_package_config_file( + ${PROJECT_SOURCE_DIR}/cmake/chai-config.cmake.in + ${PROJECT_BINARY_DIR}/chai-config.cmake + INSTALL_DESTINATION lib/cmake/chai) install( - FILES ${PROJECT_BINARY_DIR}/share/chai/cmake/chai-config.cmake - DESTINATION share/chai/cmake/) + FILES ${PROJECT_BINARY_DIR}/chai-config.cmake + DESTINATION lib/cmake/chai) + +write_basic_package_version_file( + ${PROJECT_BINARY_DIR}/chai-config-version.cmake + COMPATIBILITY SameMajorVersion) + +install(FILES + ${PROJECT_BINARY_DIR}/chai-config-version.cmake + DESTINATION lib/cmake/chai) install( FILES ${PROJECT_BINARY_DIR}/include/chai/config.hpp @@ -114,4 +116,6 @@ install( LIBRARY DESTINATION lib ARCHIVE DESTINATION lib) -install(EXPORT chai-targets DESTINATION share/chai/cmake/) +install(EXPORT chai-targets DESTINATION lib/cmake/chai) + +blt_install_tpl_setups(DESTINATION lib/cmake/chai) diff --git a/src/chai/ChaiMacros.hpp b/src/chai/ChaiMacros.hpp index 809106b9..2ff13c72 100644 --- a/src/chai/ChaiMacros.hpp +++ b/src/chai/ChaiMacros.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/ExecutionSpaces.hpp b/src/chai/ExecutionSpaces.hpp index 43346160..bdc56492 100644 --- a/src/chai/ExecutionSpaces.hpp +++ b/src/chai/ExecutionSpaces.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/ManagedArray.hpp b/src/chai/ManagedArray.hpp index e8c10ca3..6cfd5063 100644 --- a/src/chai/ManagedArray.hpp +++ b/src/chai/ManagedArray.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -81,7 +81,7 @@ class ManagedArray : public CHAICopyable /*! * \brief Default constructor creates a ManagedArray with no allocations. */ - CHAI_HOST_DEVICE ManagedArray( + ManagedArray( std::initializer_list spaces, std::initializer_list allocators); @@ -98,7 +98,7 @@ class ManagedArray : public CHAICopyable */ CHAI_HOST_DEVICE ManagedArray(size_t elems, ExecutionSpace space = get_default_space()); - CHAI_HOST_DEVICE ManagedArray( + ManagedArray( size_t elems, std::initializer_list spaces, std::initializer_list allocators, @@ -273,7 +273,7 @@ class ManagedArray : public CHAICopyable CHAI_HOST_DEVICE ManagedArray(T* data, ArrayManager* array_manager, - size_t m_elems, + size_t elems, PointerRecord* pointer_record); ManagedArray& operator=(ManagedArray const & other) = default; @@ -283,11 +283,11 @@ class ManagedArray : public CHAICopyable CHAI_HOST_DEVICE ManagedArray& operator=(std::nullptr_t); - CHAI_HOST_DEVICE bool operator==(ManagedArray& rhs) const; - CHAI_HOST_DEVICE bool operator!=(ManagedArray& from) const; + CHAI_HOST_DEVICE bool operator==(const ManagedArray& rhs) const; + CHAI_HOST_DEVICE bool operator!=(const ManagedArray& from) const; - CHAI_HOST_DEVICE bool operator==(T* from) const; - CHAI_HOST_DEVICE bool operator!=(T* from) const; + CHAI_HOST_DEVICE bool operator==(const T* from) const; + CHAI_HOST_DEVICE bool operator!=(const T* from) const; CHAI_HOST_DEVICE bool operator==(std::nullptr_t from) const; CHAI_HOST_DEVICE bool operator!=(std::nullptr_t from) const; @@ -411,16 +411,15 @@ class ManagedArray : public CHAICopyable m_active_pointer = other.m_active_pointer; m_active_base_pointer = other.m_active_base_pointer; m_resource_manager = other.m_resource_manager; - m_elems = other.m_elems; + m_size = other.m_size; m_offset = other.m_offset; m_pointer_record = other.m_pointer_record; m_is_slice = other.m_is_slice; #ifndef CHAI_DISABLE_RM #if !defined(CHAI_DEVICE_COMPILE) - // if we can, ensure elems is based off the pointer_record size to protect against - // casting leading to incorrect size info in m_elems. + // if we can, ensure elems is based off the pointer_record size out of paranoia if (m_pointer_record != nullptr && !m_is_slice) { - m_elems = m_pointer_record->m_size / sizeof(T); + m_size = m_pointer_record->m_size; } #endif #endif @@ -444,7 +443,7 @@ class ManagedArray : public CHAICopyable typename std::enable_if::type = 0> CHAI_HOST bool initInner(size_t start = 0) { - for (size_t i = start; i < m_elems; ++i) { + for (size_t i = start; i < m_size/sizeof(T); ++i) { m_active_base_pointer[i] = nullptr; } return true; @@ -473,7 +472,7 @@ class ManagedArray : public CHAICopyable /*! * Number of elements in the ManagedArray. */ - mutable size_t m_elems = 0; + mutable size_t m_size = 0; mutable size_t m_offset = 0; /*! @@ -582,7 +581,7 @@ CHAI_INLINE CHAI_HOST_DEVICE ManagedArray ManagedArray::slice( size_t offs slice.m_active_base_pointer = m_active_base_pointer; slice.m_offset = offset + m_offset; slice.m_active_pointer = m_active_base_pointer + slice.m_offset; - slice.m_elems = elems; + slice.m_size = elems*sizeof(T); slice.m_is_slice = true; } return slice; diff --git a/src/chai/ManagedArray.inl b/src/chai/ManagedArray.inl index 99ed2f44..69b9bf63 100644 --- a/src/chai/ManagedArray.inl +++ b/src/chai/ManagedArray.inl @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -18,7 +18,7 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(): m_active_pointer(nullptr), m_active_base_pointer(nullptr), m_resource_manager(nullptr), - m_elems(0), + m_size(0), m_offset(0), m_pointer_record(nullptr), m_is_slice(false) @@ -31,12 +31,11 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(): template CHAI_INLINE -CHAI_HOST_DEVICE ManagedArray::ManagedArray( +ManagedArray::ManagedArray( std::initializer_list spaces, std::initializer_list allocators): ManagedArray() { -#if !defined(CHAI_DEVICE_COMPILE) m_pointer_record = new PointerRecord(); int i = 0; for (int s = CPU; s < NUM_EXECUTION_SPACES; ++s) { @@ -46,7 +45,6 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray( for (const auto& space : spaces) { m_pointer_record->m_allocators[space] = allocators.begin()[i++].getId(); } -#endif } @@ -64,16 +62,14 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray( template CHAI_INLINE -CHAI_HOST_DEVICE ManagedArray::ManagedArray( +ManagedArray::ManagedArray( size_t elems, std::initializer_list spaces, std::initializer_list allocators, ExecutionSpace space): ManagedArray(spaces, allocators) { -#if !defined(CHAI_DEVICE_COMPILE) this->allocate(elems, space); -#endif } template @@ -89,7 +85,7 @@ CHAI_HOST ManagedArray::ManagedArray(PointerRecord* record, ExecutionSpace sp m_active_pointer(static_cast(record->m_pointers[space])), m_active_base_pointer(static_cast(record->m_pointers[space])), m_resource_manager(nullptr), - m_elems(record->m_size/sizeof(T)), + m_size(record->m_size), m_offset(0), m_pointer_record(record), m_is_slice(false) @@ -107,16 +103,16 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(ManagedArray const& other): m_active_pointer(other.m_active_pointer), m_active_base_pointer(other.m_active_base_pointer), m_resource_manager(other.m_resource_manager), - m_elems(other.m_elems), + m_size(other.m_size), m_offset(other.m_offset), m_pointer_record(other.m_pointer_record), m_is_slice(other.m_is_slice) { #if !defined(CHAI_DEVICE_COMPILE) - if (m_active_base_pointer || m_elems > 0 ) { - // we only update m_elems if we are not null and we have a pointer record + if (m_active_base_pointer || m_size > 0 ) { + // we only update m_size if we are not null and we have a pointer record if (m_pointer_record && !m_is_slice) { - m_elems = m_pointer_record->m_size/sizeof(T); + m_size = m_pointer_record->m_size; } move(m_resource_manager->getExecutionSpace()); } @@ -129,7 +125,7 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(T* data, ArrayManager* array_mana m_active_pointer(data), m_active_base_pointer(data), m_resource_manager(array_manager), - m_elems(elems), + m_size(elems*sizeof(T)), m_offset(0), m_pointer_record(pointer_record), m_is_slice(false) @@ -139,7 +135,7 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(T* data, ArrayManager* array_mana m_resource_manager = ArrayManager::getInstance(); } if (m_pointer_record == &ArrayManager::s_null_record || m_pointer_record==nullptr) { - m_pointer_record = m_resource_manager->makeManaged((void *) data, sizeof(T)*m_elems,ExecutionSpace(CPU),true); + m_pointer_record = m_resource_manager->makeManaged((void *) data, m_size,ExecutionSpace(CPU),true); } registerTouch(CPU); #endif @@ -166,8 +162,8 @@ CHAI_HOST void ManagedArray::allocate( } m_pointer_record->m_user_callback = cback; - m_elems = elems; - m_pointer_record->m_size = sizeof(T)*elems; + m_size = elems*sizeof(T); + m_pointer_record->m_size = m_size; if (space != NONE) { m_resource_manager->allocate(m_pointer_record, space); @@ -217,16 +213,16 @@ CHAI_HOST void ManagedArray::reallocate(size_t elems) { if(!m_is_slice) { if (elems > 0) { - if (m_elems == 0 && m_active_base_pointer == nullptr) { + if (m_size == 0 && m_active_base_pointer == nullptr) { return allocate(elems, CPU); } - CHAI_LOG(Debug, "Reallocating array of size " << m_elems << " with new size" << elems); + CHAI_LOG(Debug, "Reallocating array of size " << m_size << " bytes with new size" << elems*sizeof(T) << "bytes."); if (m_pointer_record == &ArrayManager::s_null_record) { - m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_elems*sizeof(T),CPU,true); + m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_size,CPU,true); } - size_t old_size = m_elems; + size_t old_size = m_size; - m_elems = elems; + m_size = elems*sizeof(T); m_active_base_pointer = static_cast(m_resource_manager->reallocate(m_active_base_pointer, elems, m_pointer_record)); @@ -235,13 +231,13 @@ CHAI_HOST void ManagedArray::reallocate(size_t elems) // if T is a CHAICopyable, then it is important to initialize all the new // ManagedArrays to nullptr at allocation, since it is extremely easy to // trigger a moveInnerImpl, which expects inner values to be initialized. - if (initInner(old_size)) { + if (initInner(old_size/sizeof(T))) { // if we are active on the GPU, we need to send any newly initialized inner members to the device - if (m_pointer_record->m_last_space == GPU && old_size < m_elems) { + if (m_pointer_record->m_last_space == GPU && old_size < m_size) { umpire::ResourceManager & umpire_rm = umpire::ResourceManager::getInstance(); - void *src = (T*)m_pointer_record->m_pointers[CPU] + old_size; - void *dst = (T*)m_pointer_record->m_pointers[GPU] + old_size; - umpire_rm.copy(dst,src,(m_elems-old_size)*sizeof(T)); + void *src = (void *)(((char *)(m_pointer_record->m_pointers[CPU])) + old_size); + void *dst = (void *)(((char *)(m_pointer_record->m_pointers[GPU])) + old_size); + umpire_rm.copy(dst,src,m_size-old_size); } } @@ -262,13 +258,13 @@ CHAI_HOST void ManagedArray::free(ExecutionSpace space) m_resource_manager = ArrayManager::getInstance(); } if (m_pointer_record == &ArrayManager::s_null_record) { - m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_elems*sizeof(T),space,true); + m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_size,space,true); } m_resource_manager->free(m_pointer_record, space); m_active_pointer = nullptr; m_active_base_pointer = nullptr; - m_elems = 0; + m_size = 0; m_offset = 0; // The call to m_resource_manager::free, above, has deallocated m_pointer_record if space == NONE. if (space == NONE) { @@ -289,7 +285,7 @@ CHAI_HOST void ManagedArray::reset() template CHAI_INLINE CHAI_HOST_DEVICE size_t ManagedArray::size() const { - return m_elems; + return m_size/sizeof(T); } template @@ -297,7 +293,7 @@ CHAI_INLINE CHAI_HOST void ManagedArray::registerTouch(ExecutionSpace space) { if (m_active_pointer && (m_pointer_record == nullptr || m_pointer_record == &ArrayManager::s_null_record)) { CHAI_LOG(Warning,"registerTouch called on ManagedArray with nullptr pointer record."); - m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_elems*sizeof(T),space,true); + m_pointer_record = m_resource_manager->makeManaged((void *)m_active_base_pointer,m_size,space,true); } m_resource_manager->registerTouch(m_pointer_record, space); } @@ -406,10 +402,14 @@ void ManagedArray::move(ExecutionSpace space, bool registerTouch) const CHAI_LOG(Debug, "Moved to " << m_active_pointer); #if defined(CHAI_ENABLE_UM) if (m_pointer_record->m_last_space == UM) { + // just because we were allocated in UM doesn't mean our CHAICopyable array values were + moveInnerImpl(); } else #endif #if defined(CHAI_ENABLE_PINNED) if (m_pointer_record->m_last_space == PINNED) { + // just because we were allocated in PINNED doesn't mean our CHAICopyable array values were + moveInnerImpl(); } else #endif if (registerTouch) { @@ -446,12 +446,12 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(T* data, CHAIDISAMBIGUATE, bool ) m_active_base_pointer(data), #if !defined(CHAI_DEVICE_COMPILE) m_resource_manager(ArrayManager::getInstance()), - m_elems(m_resource_manager->getSize((void *)m_active_base_pointer)/sizeof(T)), + m_size(m_resource_manager->getSize((void *)m_active_base_pointer)), m_offset(0), m_pointer_record(m_resource_manager->getPointerRecord((void *)data)), #else m_resource_manager(nullptr), - m_elems(0), + m_size(0), m_offset(0), m_pointer_record(nullptr), #endif @@ -493,7 +493,7 @@ T* ManagedArray::data() const { move(CPU); } - if (m_elems == 0 && !m_is_slice) { + if (m_size == 0 && !m_is_slice) { return nullptr; } @@ -516,7 +516,7 @@ const T* ManagedArray::cdata() const { move(CPU, false); } - if (m_elems == 0 && !m_is_slice) { + if (m_size == 0 && !m_is_slice) { return nullptr; } @@ -532,7 +532,7 @@ T* ManagedArray::data(ExecutionSpace space, bool do_move) const { return nullptr; } - if (m_elems == 0 && !m_is_slice) { + if (m_size == 0 && !m_is_slice) { return nullptr; } @@ -596,7 +596,7 @@ ManagedArray& ManagedArray::operator= (std::nullptr_t) { m_active_pointer = nullptr; m_active_base_pointer = nullptr; - m_elems = 0; + m_size = 0; m_offset = 0; #if !defined(CHAI_DEVICE_COMPILE) m_pointer_record = &ArrayManager::s_null_record; @@ -613,7 +613,7 @@ template CHAI_INLINE CHAI_HOST_DEVICE bool -ManagedArray::operator== (ManagedArray& rhs) const +ManagedArray::operator== (const ManagedArray& rhs) const { return (m_active_pointer == rhs.m_active_pointer); } @@ -622,7 +622,7 @@ template CHAI_INLINE CHAI_HOST_DEVICE bool -ManagedArray::operator!= (ManagedArray& rhs) const +ManagedArray::operator!= (const ManagedArray& rhs) const { return (m_active_pointer != rhs.m_active_pointer); } @@ -632,7 +632,7 @@ template CHAI_INLINE CHAI_HOST_DEVICE bool -ManagedArray::operator== (T * from) const { +ManagedArray::operator== (const T * from) const { return m_active_pointer == from; } @@ -640,7 +640,7 @@ template CHAI_INLINE CHAI_HOST_DEVICE bool -ManagedArray::operator!= (T * from) const { +ManagedArray::operator!= (const T * from) const { return m_active_pointer != from; } @@ -649,21 +649,21 @@ CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator== (std::nullptr_t from) const { - return m_active_pointer == from || m_elems == 0; + return m_active_pointer == from || m_size == 0; } template CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator!= (std::nullptr_t from) const { - return m_active_pointer != from && m_elems > 0; + return m_active_pointer != from && m_size > 0; } template CHAI_INLINE CHAI_HOST_DEVICE ManagedArray::operator bool () const { - return m_elems > 0; + return m_size > 0; } template diff --git a/src/chai/ManagedArrayView.hpp b/src/chai/ManagedArrayView.hpp index 85ee8407..4791931f 100644 --- a/src/chai/ManagedArrayView.hpp +++ b/src/chai/ManagedArrayView.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/ManagedArray_thin.inl b/src/chai/ManagedArray_thin.inl index 586c7245..d9e504e5 100644 --- a/src/chai/ManagedArray_thin.inl +++ b/src/chai/ManagedArray_thin.inl @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -10,13 +10,15 @@ #include "ManagedArray.hpp" #if defined(CHAI_ENABLE_UM) +#if !defined(CHAI_THIN_GPU_ALLOCATE) #include #endif +#endif namespace chai { template -CHAI_INLINE CHAI_HOST_DEVICE ManagedArray::ManagedArray( +CHAI_INLINE ManagedArray::ManagedArray( std::initializer_list spaces, std::initializer_list allocators) : ManagedArray() @@ -32,14 +34,14 @@ CHAI_INLINE CHAI_HOST_DEVICE ManagedArray::ManagedArray( template CHAI_INLINE -CHAI_HOST_DEVICE ManagedArray::ManagedArray( +ManagedArray::ManagedArray( size_t elems, std::initializer_list spaces, std::initializer_list allocators, ExecutionSpace space) : ManagedArray(spaces, allocators) { - m_elems = elems; + m_size = elems*sizeof(T); this->allocate(elems, space); } @@ -50,9 +52,11 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray() = default; template CHAI_INLINE CHAI_HOST_DEVICE ManagedArray::ManagedArray(size_t elems, ExecutionSpace space) : - m_elems(elems) + m_size(elems*sizeof(T)) { +#ifndef CHAI_DEVICE_COMPILE this->allocate(elems, space); +#endif } @@ -61,7 +65,7 @@ CHAI_INLINE CHAI_HOST_DEVICE ManagedArray::ManagedArray(std::nullptr_t) : m_active_pointer(nullptr), m_active_base_pointer(nullptr), m_resource_manager(nullptr), - m_elems(0), + m_size(0), m_offset(0), m_pointer_record(nullptr), m_is_slice(false) @@ -75,7 +79,7 @@ CHAI_HOST ManagedArray::ManagedArray(PointerRecord* record, ExecutionSpace sp m_active_pointer(static_cast(record->m_pointers[space])), m_active_base_pointer(static_cast(record->m_pointers[space])), m_resource_manager(nullptr), - m_elems(record->m_size/sizeof(T)), + m_size(record->m_size), m_offset(0), m_pointer_record(nullptr), m_is_slice(!record->m_owned[space]) @@ -95,19 +99,19 @@ CHAI_HOST_DEVICE ManagedArray::ManagedArray(T* data, m_active_pointer(data), m_active_base_pointer(data), m_resource_manager(array_manager), - m_elems(elems), + m_size(elems*sizeof(T)), m_pointer_record(pointer_record) { } template -T* ManagedArray::getActiveBasePointer() const +CHAI_HOST_DEVICE T* ManagedArray::getActiveBasePointer() const { return m_active_base_pointer; } template -T* ManagedArray::getActivePointer() const +CHAI_HOST_DEVICE T* ManagedArray::getActivePointer() const { return m_active_pointer; } @@ -115,18 +119,29 @@ T* ManagedArray::getActivePointer() const template CHAI_HOST_DEVICE T* ManagedArray::data() const { +#if !defined(CHAI_DEVICE_COMPILE) && defined(CHAI_THIN_GPU_ALLOCATE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif return m_active_pointer; } template CHAI_HOST_DEVICE const T* ManagedArray::cdata() const { +#if !defined(CHAI_DEVICE_COMPILE) && defined(CHAI_THIN_GPU_ALLOCATE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif return m_active_pointer; } template -T* ManagedArray::data(ExecutionSpace /*space*/, bool /*do_move*/) const +T* ManagedArray::data(ExecutionSpace space, bool do_move) const { +#if defined(CHAI_THIN_GPU_ALLOCATE) + if (do_move && space != chai::GPU) { + ArrayManager::getInstance()->syncIfNeeded(); + } +#endif return m_active_pointer; } @@ -160,10 +175,12 @@ CHAI_HOST void ManagedArray::allocate(size_t elems, << " in space " << space); - m_elems = elems; + m_size = elems*sizeof(T); - #if defined(CHAI_ENABLE_UM) - gpuMallocManaged(&m_active_pointer, sizeof(T) * elems); + #if defined(CHAI_THIN_GPU_ALLOCATE) + m_active_pointer = (T*) chai::ArrayManager::getInstance()->getAllocator(chai::GPU).allocate(m_size); + #elif defined(CHAI_ENABLE_UM) + gpuMallocManaged(&m_active_pointer, m_size); #else // not CHAI_ENABLE_UM m_active_pointer = static_cast(malloc(sizeof(T) * elems)); #endif @@ -173,7 +190,7 @@ CHAI_HOST void ManagedArray::allocate(size_t elems, } else { m_active_pointer = nullptr; - m_elems = 0; + m_size = 0; } } else { @@ -187,21 +204,37 @@ CHAI_INLINE CHAI_HOST void ManagedArray::reallocate(size_t new_elems) { if (!m_is_slice) { - CHAI_LOG(Debug, "Reallocating array of size " << m_elems + CHAI_LOG(Debug, "Reallocating array of size " << m_size*sizeof(T) << " with new size" - << elems); + << new_elems*sizeof(T)); - T* new_ptr; + T* new_ptr = nullptr; - #if defined(CHAI_ENABLE_UM) - gpuMallocManaged(&new_ptr, sizeof(T) * new_elems); - gpuMemcpy(new_ptr, m_active_pointer, sizeof(T) * m_elems, gpuMemcpyDefault); + #if defined(CHAI_THIN_GPU_ALLOCATE) + auto allocator = chai::ArrayManager::getInstance()->getAllocator(chai::GPU); + if (new_elems > 0) { + new_ptr = (T*) allocator.allocate(sizeof(T) * new_elems); + ArrayManager::getInstance()->syncIfNeeded(); + chai::gpuMemcpy(new_ptr, m_active_pointer, std::min(m_size, new_elems*sizeof(T)), gpuMemcpyDefault); + registerTouch(chai::GPU); + } + allocator.deallocate(m_active_pointer); + #elif defined(CHAI_ENABLE_UM) + if (new_elems > 0) { + gpuMallocManaged(&new_ptr, sizeof(T) * new_elems); + gpuMemcpy(new_ptr, m_active_pointer, std::min(new_elems*sizeof(T), m_size), gpuMemcpyDefault); + } gpuFree(m_active_pointer); #else // not CHAI_ENABLE_UM - new_ptr = static_cast(realloc(m_active_pointer, sizeof(T) * new_elems)); + if (new_elems > 0) { + new_ptr = static_cast(realloc(m_active_pointer, sizeof(T) * new_elems)); + } + else { + ::free((void *)m_active_pointer); + } #endif - m_elems = new_elems; + m_size= new_elems*sizeof(T); m_active_pointer = new_ptr; m_active_base_pointer = m_active_pointer; @@ -217,14 +250,19 @@ CHAI_INLINE CHAI_HOST void ManagedArray::free(ExecutionSpace space) { if (!m_is_slice) { if (space == CPU || space == NONE) { -#if defined(CHAI_ENABLE_UM) - gpuFree(m_active_pointer); +#if defined(CHAI_THIN_GPU_ALLOCATE) + if (m_active_pointer) { + auto allocator = chai::ArrayManager::getInstance()->getAllocator(chai::GPU); + allocator.deallocate((void *)m_active_pointer); + } +#elif defined(CHAI_ENABLE_UM) + chai::gpuFree(m_active_pointer); #else ::free((void *)m_active_pointer); #endif m_active_pointer = nullptr; m_active_base_pointer = nullptr; - m_elems = 0; + m_size = 0; } } else { @@ -244,7 +282,11 @@ template CHAI_INLINE CHAI_HOST_DEVICE typename ManagedArray::T_non_const ManagedArray< T>::pick(size_t i) const { -#ifdef CHAI_ENABLE_UM +#if defined(CHAI_THIN_GPU_ALLOCATE) +#if !defined(CHAI_DEVICE_COMPILE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif +#elif defined(CHAI_ENABLE_UM) synchronize(); #endif return (T_non_const)m_active_pointer[i]; @@ -253,7 +295,11 @@ CHAI_INLINE CHAI_HOST_DEVICE typename ManagedArray::T_non_const ManagedArray< template CHAI_INLINE CHAI_HOST_DEVICE void ManagedArray::set(size_t i, T val) const { -#if defined(CHAI_ENABLE_UM) +#if defined(CHAI_THIN_GPU_ALLOCATE) +#if !defined(CHAI_DEVICE_COMPILE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif +#elif defined(CHAI_ENABLE_UM) synchronize(); #endif m_active_pointer[i] = val; @@ -262,7 +308,11 @@ CHAI_INLINE CHAI_HOST_DEVICE void ManagedArray::set(size_t i, T val) const template CHAI_INLINE CHAI_HOST_DEVICE void ManagedArray::incr(size_t i) const { -#if defined(CHAI_ENABLE_UM) +#if defined(CHAI_THIN_GPU_ALLOCATE) +#if !defined(CHAI_DEVICE_COMPILE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif +#elif defined(CHAI_ENABLE_UM) synchronize(); #endif ++m_active_pointer[i]; @@ -271,22 +321,29 @@ CHAI_INLINE CHAI_HOST_DEVICE void ManagedArray::incr(size_t i) const template CHAI_INLINE CHAI_HOST_DEVICE void ManagedArray::decr(size_t i) const { -#if defined(CHAI_ENABLE_UM) +#if defined(CHAI_THIN_GPU_ALLOCATE) +#if !defined(CHAI_DEVICE_COMPILE) + ArrayManager::getInstance()->syncIfNeeded(); +#endif +#elif defined(CHAI_ENABLE_UM) synchronize(); #endif --m_active_pointer[i]; } -#endif +#endif // CHAI_ENABLE_PICK template CHAI_INLINE CHAI_HOST_DEVICE size_t ManagedArray::size() const { - return m_elems; + return m_size/sizeof(T); } template -CHAI_INLINE CHAI_HOST void ManagedArray::registerTouch(ExecutionSpace) +CHAI_INLINE CHAI_HOST void ManagedArray::registerTouch(ExecutionSpace space) { +#if defined(CHAI_THIN_GPU_ALLOCATE) + chai::ArrayManager::getInstance()->setExecutionSpace(space) ; +#endif } template @@ -315,7 +372,7 @@ ManagedArray::ManagedArray(T* data, CHAIDISAMBIGUATE, bool) : m_active_pointer(data), m_active_base_pointer(data), m_resource_manager(nullptr), - m_elems(-1), + m_size(-1), m_pointer_record(nullptr), m_offset(0), m_is_slice(false) @@ -330,7 +387,7 @@ ManagedArray::operator typename std:: { return ManagedArray(const_cast(m_active_pointer), m_resource_manager, - m_elems, + m_size/sizeof(T), nullptr); } @@ -351,34 +408,34 @@ CHAI_INLINE CHAI_HOST_DEVICE ManagedArray& ManagedArray::operator=(std::nu { m_active_pointer = from; m_active_base_pointer = from; - m_elems = 0; + m_size = 0; m_is_slice = false; return *this; } template CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator==( - ManagedArray& rhs) const + const ManagedArray& rhs) const { return (m_active_pointer == rhs.m_active_pointer); } template CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator!=( - ManagedArray& rhs) const + const ManagedArray& rhs) const { return (m_active_pointer != rhs.m_active_pointer); } template -CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator==(T* from) const +CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator==(const T* from) const { return m_active_pointer == from; } template -CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator!=(T* from) const +CHAI_INLINE CHAI_HOST_DEVICE bool ManagedArray::operator!=(const T* from) const { return m_active_pointer != from; } diff --git a/src/chai/PointerRecord.hpp b/src/chai/PointerRecord.hpp index 9ff9c372..e46ea899 100644 --- a/src/chai/PointerRecord.hpp +++ b/src/chai/PointerRecord.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/RajaExecutionSpacePlugin.cpp b/src/chai/RajaExecutionSpacePlugin.cpp index e914d7f3..42db1e66 100644 --- a/src/chai/RajaExecutionSpacePlugin.cpp +++ b/src/chai/RajaExecutionSpacePlugin.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/RajaExecutionSpacePlugin.hpp b/src/chai/RajaExecutionSpacePlugin.hpp index b9ccc36e..9ea3cbce 100644 --- a/src/chai/RajaExecutionSpacePlugin.hpp +++ b/src/chai/RajaExecutionSpacePlugin.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/Types.hpp b/src/chai/Types.hpp index 0164a2e0..300bb12c 100644 --- a/src/chai/Types.hpp +++ b/src/chai/Types.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/chai-config.cmake.in b/src/chai/chai-config.cmake.in deleted file mode 100644 index 4fc7af6a..00000000 --- a/src/chai/chai-config.cmake.in +++ /dev/null @@ -1,13 +0,0 @@ -############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. -# -# SPDX-License-Identifier: BSD-3-Clause -############################################################################## - -set (CHAI_INSTALL_PREFIX @CMAKE_INSTALL_PREFIX@) -set (CHAI_INCLUDE_DIRS @CMAKE_INSTALL_PREFIX@/include) -set (CHAI_LIB_DIR @CMAKE_INSTALL_PREFIX@/lib) -set (CHAI_CMAKE_DIR @CMAKE_INSTALL_PREFIX@/share/chai/cmake) - -include(@CMAKE_INSTALL_PREFIX@/share/chai/cmake/chai-targets.cmake) diff --git a/src/chai/config.hpp.in b/src/chai/config.hpp.in index 8325184c..fdb77cdd 100644 --- a/src/chai/config.hpp.in +++ b/src/chai/config.hpp.in @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -12,10 +12,12 @@ #cmakedefine CHAI_ENABLE_HIP #cmakedefine CHAI_ENABLE_IMPLICIT_CONVERSIONS #cmakedefine CHAI_DISABLE_RM +#cmakedefine CHAI_THIN_GPU_ALLOCATE #cmakedefine CHAI_ENABLE_UM #cmakedefine CHAI_DEBUG #cmakedefine CHAI_ENABLE_GPU_ERROR_CHECKING #cmakedefine CHAI_ENABLE_MANAGED_PTR +#cmakedefine CHAI_ENABLE_MANAGED_PTR_ON_GPU #cmakedefine CHAI_ENABLE_RAJA_PLUGIN #cmakedefine CHAI_ENABLE_GPU_SIMULATION_MODE #cmakedefine CHAI_ENABLE_PINNED diff --git a/src/chai/managed_array.hpp b/src/chai/managed_array.hpp index 2193e54c..f21056a7 100644 --- a/src/chai/managed_array.hpp +++ b/src/chai/managed_array.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/chai/managed_ptr.hpp b/src/chai/managed_ptr.hpp index 55a0f122..14988cb0 100644 --- a/src/chai/managed_ptr.hpp +++ b/src/chai/managed_ptr.hpp @@ -1,17 +1,17 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// -#ifndef MANAGED_PTR_H_ -#define MANAGED_PTR_H_ +#ifndef CHAI_MANAGED_PTR_HPP +#define CHAI_MANAGED_PTR_HPP #include "chai/config.hpp" #if defined(CHAI_ENABLE_MANAGED_PTR) -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) || defined(CHAI_THIN_GPU_ALLOCATE) #include "chai/ArrayManager.hpp" #endif @@ -26,12 +26,10 @@ namespace chai { - namespace detail { -#if defined(CHAI_GPUCC) - template - __global__ void destroy_on_device(T* gpuPointer); +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + template + CHAI_HOST void destroy_on_device(T* gpuPointer); #endif - } struct managed_ptr_record { managed_ptr_record() = default; @@ -103,7 +101,7 @@ namespace chai { /// Be aware that CHAI checks every CUDA API call for GPU errors by default. To /// turn off GPU error checking, pass -DCHAI_ENABLE_GPU_ERROR_CHECKING=OFF as /// an argument to cmake when building CHAI. To turn on synchronization after - /// every kernel, call ArrayManager::getInstance()->enableDeviceSynchronize(). + /// every kernel, set the appropriate environment variable (e.g. CUDA_LAUNCH_BLOCKING or HIP_LAUNCH_BLOCKING). /// Alternatively, call cudaDeviceSynchronize() after any call to make_managed, /// make_managed_from_factory, or managed_ptr::free, and check the return code /// for errors. If your code crashes in the constructor/destructor of T, then @@ -145,7 +143,9 @@ namespace chai { managed_ptr(std::initializer_list spaces, std::initializer_list pointers) : m_cpu_pointer(nullptr), +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer(nullptr), +#endif m_pointer_record(new managed_ptr_record()) { static_assert(std::is_convertible::value, @@ -163,7 +163,7 @@ namespace chai { case CPU: m_cpu_pointer = pointers.begin()[i++]; break; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: m_gpu_pointer = pointers.begin()[i++]; break; @@ -193,7 +193,9 @@ namespace chai { std::initializer_list pointers, std::function callback) : m_cpu_pointer(nullptr), +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer(nullptr), +#endif m_pointer_record(new managed_ptr_record(callback)) { static_assert(std::is_convertible::value, @@ -211,7 +213,7 @@ namespace chai { case CPU: m_cpu_pointer = pointers.begin()[i++]; break; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: m_gpu_pointer = pointers.begin()[i++]; break; @@ -237,7 +239,9 @@ namespace chai { /// CHAI_HOST_DEVICE managed_ptr(const managed_ptr& other) noexcept : m_cpu_pointer(other.m_cpu_pointer), +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer(other.m_gpu_pointer), +#endif m_pointer_record(other.m_pointer_record) { #if !defined(CHAI_DEVICE_COMPILE) @@ -259,7 +263,9 @@ namespace chai { template CHAI_HOST_DEVICE managed_ptr(const managed_ptr& other) noexcept : m_cpu_pointer(other.m_cpu_pointer), +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer(other.m_gpu_pointer), +#endif m_pointer_record(other.m_pointer_record) { static_assert(std::is_convertible::value, @@ -300,7 +306,7 @@ namespace chai { case CPU: m_cpu_pointer = pointers.begin()[i++]; break; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: m_gpu_pointer = pointers.begin()[i++]; break; @@ -328,7 +334,9 @@ namespace chai { CHAI_HOST_DEVICE managed_ptr& operator=(const managed_ptr& other) noexcept { if (this != &other) { m_cpu_pointer = other.m_cpu_pointer; +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer = other.m_gpu_pointer; +#endif m_pointer_record = other.m_pointer_record; #if !defined(CHAI_DEVICE_COMPILE) @@ -356,7 +364,9 @@ namespace chai { "U* must be convertible to T*."); m_cpu_pointer = other.m_cpu_pointer; +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) m_gpu_pointer = other.m_gpu_pointer; +#endif m_pointer_record = other.m_pointer_record; #if !defined(CHAI_DEVICE_COMPILE) @@ -372,11 +382,13 @@ namespace chai { /// Returns the CPU or GPU pointer depending on the calling context. /// CHAI_HOST_DEVICE inline T* get() const { +#if defined(CHAI_DEVICE_COMPILE) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + return m_gpu_pointer; +#else #if !defined(CHAI_DEVICE_COMPILE) move(); +#endif return m_cpu_pointer; -#else - return m_gpu_pointer; #endif } @@ -396,7 +408,7 @@ namespace chai { switch (space) { case CPU: return m_cpu_pointer; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: return m_gpu_pointer; #endif @@ -411,10 +423,13 @@ namespace chai { /// Returns the CPU or GPU pointer depending on the calling context. /// CHAI_HOST_DEVICE inline T* operator->() const { +#if defined(CHAI_DEVICE_COMPILE) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + return m_gpu_pointer; +#else #if !defined(CHAI_DEVICE_COMPILE) + move(); +#endif return m_cpu_pointer; -#else - return m_gpu_pointer; #endif } @@ -424,10 +439,13 @@ namespace chai { /// Returns the CPU or GPU reference depending on the calling context. /// CHAI_HOST_DEVICE inline T& operator*() const { +#if defined(CHAI_DEVICE_COMPILE) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + return *m_gpu_pointer; +#else #if !defined(CHAI_DEVICE_COMPILE) + move(); +#endif return *m_cpu_pointer; -#else - return *m_gpu_pointer; #endif } @@ -498,18 +516,12 @@ namespace chai { delete pointer; m_cpu_pointer = nullptr; break; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: { if (pointer) { - detail::destroy_on_device<<<1, 1>>>(temp); + destroy_on_device(temp); m_gpu_pointer = nullptr; - -#ifndef CHAI_DISABLE_RM - if (ArrayManager::getInstance()->deviceSynchronize()) { - synchronize(); - } -#endif } break; @@ -532,18 +544,12 @@ namespace chai { delete pointer; m_cpu_pointer = nullptr; break; -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) case GPU: { if (pointer) { - detail::destroy_on_device<<<1, 1>>>(pointer); + destroy_on_device(pointer); m_gpu_pointer = nullptr; - -#ifndef CHAI_DISABLE_RM - if (ArrayManager::getInstance()->deviceSynchronize()) { - synchronize(); - } -#endif } break; @@ -562,7 +568,9 @@ namespace chai { private: T* m_cpu_pointer = nullptr; /// The CPU pointer +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) T* m_gpu_pointer = nullptr; /// The GPU pointer +#endif managed_ptr_record* m_pointer_record = nullptr; /// The pointer record /// Needed for the converting constructor @@ -581,7 +589,7 @@ namespace chai { /// with the ACTION_MOVE event. /// CHAI_HOST void move() const { -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) if (m_pointer_record) { ExecutionSpace newSpace = ArrayManager::getInstance()->getExecutionSpace(); @@ -734,7 +742,7 @@ namespace chai { return arg.get(); } -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) /// /// @author Alan Dayton @@ -861,7 +869,7 @@ namespace chai { template CHAI_HOST T* make_on_host(Args&&... args) { -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Get the ArrayManager and save the current execution space chai::ArrayManager* arrayManager = chai::ArrayManager::getInstance(); ExecutionSpace currentSpace = arrayManager->getExecutionSpace(); @@ -874,7 +882,7 @@ namespace chai { // Create on the host T* cpuPointer = new T(detail::processArguments(args)...); -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Set the execution space back to the previous value arrayManager->setExecutionSpace(currentSpace); #endif @@ -899,7 +907,7 @@ namespace chai { typename F, typename... Args> CHAI_HOST T* make_on_host_from_factory(F f, Args&&... args) { -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Get the ArrayManager and save the current execution space chai::ArrayManager* arrayManager = chai::ArrayManager::getInstance(); ExecutionSpace currentSpace = arrayManager->getExecutionSpace(); @@ -912,7 +920,7 @@ namespace chai { // Create the object on the device T* cpuPointer = f(args...); -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Set the execution space back to the previous value arrayManager->setExecutionSpace(currentSpace); #endif @@ -933,21 +941,21 @@ namespace chai { delete cpuPointer; } -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) /// /// @author Alan Dayton /// /// Creates a new T on the device. /// - /// @param[in] args The arguments to T's constructor + /// @param[in] args The arguments to T's constructor /// /// @return The device pointer to the new T /// template CHAI_HOST T* make_on_device(Args... args) { -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Get the ArrayManager and save the current execution space chai::ArrayManager* arrayManager = chai::ArrayManager::getInstance(); ExecutionSpace currentSpace = arrayManager->getExecutionSpace(); @@ -962,12 +970,10 @@ namespace chai { gpuMalloc((void**)(&gpuBuffer), sizeof(T*)); // Create the object on the device +#if defined(__CUDACC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) detail::make_on_device<<<1, 1>>>(gpuBuffer, args...); - -#ifndef CHAI_DISABLE_RM - if (ArrayManager::getInstance()->deviceSynchronize()) { - synchronize(); - } +#elif defined(__HIPCC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + hipLaunchKernelGGL(detail::make_on_device, 1, 1, 0, 0, gpuBuffer, args...); #endif // Allocate space on the CPU for the pointer and copy the pointer to the CPU @@ -981,7 +987,7 @@ namespace chai { free(cpuBuffer); gpuFree(gpuBuffer); -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Set the execution space back to the previous value arrayManager->setExecutionSpace(currentSpace); #endif @@ -1004,7 +1010,7 @@ namespace chai { typename F, typename... Args> CHAI_HOST T* make_on_device_from_factory(F f, Args&&... args) { -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Get the ArrayManager and save the current execution space chai::ArrayManager* arrayManager = chai::ArrayManager::getInstance(); ExecutionSpace currentSpace = arrayManager->getExecutionSpace(); @@ -1019,12 +1025,10 @@ namespace chai { gpuMalloc((void**)(&gpuBuffer), sizeof(T*)); // Create the object on the device +#if defined(__CUDACC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) detail::make_on_device_from_factory<<<1, 1>>>(gpuBuffer, f, args...); - -#ifndef CHAI_DISABLE_RM - if (ArrayManager::getInstance()->deviceSynchronize()) { - synchronize(); - } +#elif defined(__HIPCC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + hipLaunchKernelGGL(detail::make_on_device_from_factory, 1, 1, 0, 0, gpuBuffer, f, args...); #endif // Allocate space on the CPU for the pointer and copy the pointer to the CPU @@ -1038,7 +1042,7 @@ namespace chai { free(cpuBuffer); gpuFree(gpuBuffer); -#ifndef CHAI_DISABLE_RM +#if !defined(CHAI_DISABLE_RM) // Set the execution space back to the previous value arrayManager->setExecutionSpace(currentSpace); #endif @@ -1056,7 +1060,11 @@ namespace chai { /// template CHAI_HOST void destroy_on_device(T* gpuPointer) { +#if defined(__CUDACC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) detail::destroy_on_device<<<1, 1>>>(gpuPointer); +#elif defined(__HIPCC__) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) + hipLaunchKernelGGL(detail::destroy_on_device, 1, 1, 0, 0, gpuPointer); +#endif } #endif @@ -1072,7 +1080,7 @@ namespace chai { template CHAI_HOST managed_ptr make_managed(Args... args) { -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) // Construct on the GPU first to take advantage of asynchrony T* gpuPointer = make_on_device(args...); #endif @@ -1081,7 +1089,7 @@ namespace chai { T* cpuPointer = make_on_host(args...); // Construct and return the managed_ptr -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) return managed_ptr({CPU, GPU}, {cpuPointer, gpuPointer}); #else return managed_ptr({CPU}, {cpuPointer}); @@ -1112,7 +1120,7 @@ namespace chai { static_assert(std::is_convertible::value, "F does not return a pointer that is convertible to T*."); -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) // Construct on the GPU first to take advantage of asynchrony T* gpuPointer = make_on_device_from_factory(f, args...); #endif @@ -1121,7 +1129,7 @@ namespace chai { T* cpuPointer = make_on_host_from_factory(f, args...); // Construct and return the managed_ptr -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) return managed_ptr({CPU, GPU}, {cpuPointer, gpuPointer}); #else return managed_ptr({CPU}, {cpuPointer}); @@ -1141,7 +1149,7 @@ namespace chai { CHAI_HOST managed_ptr static_pointer_cast(const managed_ptr& other) noexcept { T* cpuPointer = static_cast(other.get()); -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) T* gpuPointer = static_cast(other.get(GPU, false)); return managed_ptr(other, {CPU, GPU}, {cpuPointer, gpuPointer}); @@ -1163,7 +1171,7 @@ namespace chai { CHAI_HOST managed_ptr dynamic_pointer_cast(const managed_ptr& other) noexcept { T* cpuPointer = dynamic_cast(other.get()); -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) T* gpuPointer = nullptr; if (cpuPointer) { @@ -1189,7 +1197,7 @@ namespace chai { CHAI_HOST managed_ptr const_pointer_cast(const managed_ptr& other) noexcept { T* cpuPointer = const_cast(other.get()); -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) T* gpuPointer = const_cast(other.get(GPU, false)); return managed_ptr(other, {CPU, GPU}, {cpuPointer, gpuPointer}); @@ -1211,7 +1219,7 @@ namespace chai { CHAI_HOST managed_ptr reinterpret_pointer_cast(const managed_ptr& other) noexcept { T* cpuPointer = reinterpret_cast(other.get()); -#if defined(CHAI_GPUCC) +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) T* gpuPointer = reinterpret_cast(other.get(GPU, false)); return managed_ptr(other, {CPU, GPU}, {cpuPointer, gpuPointer}); @@ -1315,7 +1323,9 @@ namespace chai { template void swap(managed_ptr& lhs, managed_ptr& rhs) noexcept { std::swap(lhs.m_cpu_pointer, rhs.m_cpu_pointer); +#if defined(CHAI_GPUCC) && defined(CHAI_ENABLE_MANAGED_PTR_ON_GPU) std::swap(lhs.m_gpu_pointer, rhs.m_gpu_pointer); +#endif std::swap(lhs.m_pointer_record, rhs.m_pointer_record); } } // namespace chai diff --git a/src/chai/pluginLinker.hpp b/src/chai/pluginLinker.hpp index cb25b798..e7af9a76 100644 --- a/src/chai/pluginLinker.hpp +++ b/src/chai/pluginLinker.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/src/tpl/raja b/src/tpl/raja index 357933a4..82d1b926 160000 --- a/src/tpl/raja +++ b/src/tpl/raja @@ -1 +1 @@ -Subproject commit 357933a42842dd91de5c1034204d937fce0a2a44 +Subproject commit 82d1b926ada0fbb15a4a6e0adadc30c715cfda7b diff --git a/src/tpl/umpire b/src/tpl/umpire index 5f886b42..1db3fef9 160000 --- a/src/tpl/umpire +++ b/src/tpl/umpire @@ -1 +1 @@ -Subproject commit 5f886b4299496b7cb6f9d62dc1372ce6d3832fbc +Subproject commit 1db3fef913a70d8882ca510a4830c77c388873e0 diff --git a/src/util/forall.hpp b/src/util/forall.hpp index 38a21a5a..c90b944d 100644 --- a/src/util/forall.hpp +++ b/src/util/forall.hpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// @@ -12,8 +12,10 @@ #include "chai/config.hpp" #if defined(CHAI_ENABLE_UM) +#if !defined(CHAI_THIN_GPU_ALLOCATE) #include #endif +#endif struct sequential { }; @@ -41,7 +43,9 @@ void forall(sequential, int begin, int end, LOOP_BODY body) chai::ArrayManager* rm = chai::ArrayManager::getInstance(); #if defined(CHAI_ENABLE_UM) +#if !defined(CHAI_THIN_GPU_ALLOCATE) cudaDeviceSynchronize(); +#endif #endif rm->setExecutionSpace(chai::CPU); @@ -69,14 +73,20 @@ void forall(gpu_async, int begin, int end, LOOP_BODY&& body) rm->setExecutionSpace(chai::GPU); +#if defined(CHAI_ENABLE_CUDA) size_t blockSize = 32; +#elif defined(CHAI_ENABLE_HIP) + size_t blockSize = 64; +#endif + size_t gridSize = (end - begin + blockSize - 1) / blockSize; + #if defined(CHAI_ENABLE_GPU_SIMULATION_MODE) forall_kernel_cpu(begin, end, body); #elif defined(CHAI_ENABLE_CUDA) forall_kernel_gpu<<>>(begin, end - begin, body); #elif defined(CHAI_ENABLE_HIP) - hipLaunchKernelGGL(forall_kernel_gpu, dim3(gridSize), dim3(blockSize), 0,0, + hipLaunchKernelGGL(forall_kernel_gpu, dim3(gridSize), dim3(blockSize), 0, 0, begin, end - begin, body); #endif rm->setExecutionSpace(chai::NONE); @@ -92,7 +102,12 @@ void forall(gpu, int begin, int end, LOOP_BODY&& body) rm->setExecutionSpace(chai::GPU); +#if defined(CHAI_ENABLE_CUDA) size_t blockSize = 32; +#elif defined(CHAI_ENABLE_HIP) + size_t blockSize = 64; +#endif + size_t gridSize = (end - begin + blockSize - 1) / blockSize; #if defined(CHAI_ENABLE_GPU_SIMULATION_MODE) @@ -101,7 +116,7 @@ void forall(gpu, int begin, int end, LOOP_BODY&& body) forall_kernel_gpu<<>>(begin, end - begin, body); cudaDeviceSynchronize(); #elif defined(CHAI_ENABLE_HIP) - hipLaunchKernelGGL(forall_kernel_gpu, dim3(gridSize), dim3(blockSize), 0,0, + hipLaunchKernelGGL(forall_kernel_gpu, dim3(gridSize), dim3(blockSize), 0, 0, begin, end - begin, body); hipDeviceSynchronize(); #endif diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index b2d1cba5..97e5c4eb 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1,9 +1,10 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## -add_subdirectory(unit) +add_subdirectory(install) +add_subdirectory(unit) add_subdirectory(integration) diff --git a/tests/install/CMakeLists.txt b/tests/install/CMakeLists.txt new file mode 100644 index 00000000..7b1616d5 --- /dev/null +++ b/tests/install/CMakeLists.txt @@ -0,0 +1,16 @@ +############################################################################### +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC +# and CHAI project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (BSD-3-Clause) +############################################################################### + +configure_file( + using-with-cmake/host-config.cmake.in + ${PROJECT_BINARY_DIR}/examples/using-with-cmake/host-config.cmake) + + install( FILES + using-with-cmake/CMakeLists.txt + using-with-cmake/using-with-cmake.cpp + ${PROJECT_BINARY_DIR}/examples/using-with-cmake/host-config.cmake + DESTINATION examples/chai/using-with-cmake) diff --git a/tests/install/using-with-cmake/CMakeLists.txt b/tests/install/using-with-cmake/CMakeLists.txt new file mode 100644 index 00000000..8c43e078 --- /dev/null +++ b/tests/install/using-with-cmake/CMakeLists.txt @@ -0,0 +1,51 @@ +############################################################################### +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC +# and CHAI project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (BSD-3-Clause) +############################################################################### + +if (ENABLE_HIP) + cmake_minimum_required(VERSION 3.23) +else() + cmake_minimum_required(VERSION 3.20) +endif() + +project(using_with_cmake) + +if (ENABLE_CUDA) + enable_language(CUDA) +endif() + +if (ENABLE_HIP) + if (NOT ROCM_PATH) + find_path(ROCM_PATH + hip + ENV{ROCM_DIR} + ENV{ROCM_PATH} + ENV{HIP_PATH} + ${HIP_PATH}/.. + ${HIP_ROOT_DIR}/../ + ${ROCM_ROOT_DIR} + /opt/rocm) + endif() + + set(CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH};${ROCM_PATH}") + find_package(hip REQUIRED CONFIG PATHS ${ROCM_PATH}) +endif() + +find_package(chai CONFIG REQUIRED NO_DEFAULT_PATH PATHS ${chai_DIR} ${CHAI_DIR}) + +# Remove implicitly added link directories added by CMake that are problematic when +# the default system libraries are older than the ones used by the compiler +if (BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE) + list(REMOVE_ITEM CMAKE_C_IMPLICIT_LINK_DIRECTORIES + ${BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE}) + list(REMOVE_ITEM CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES + ${BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE}) + list(REMOVE_ITEM CMAKE_CUDA_IMPLICIT_LINK_DIRECTORIES + ${BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE}) +endif () + +add_executable(using-with-cmake using-with-cmake.cpp) +target_link_libraries(using-with-cmake chai) diff --git a/tests/install/using-with-cmake/host-config.cmake.in b/tests/install/using-with-cmake/host-config.cmake.in new file mode 100644 index 00000000..ea36a492 --- /dev/null +++ b/tests/install/using-with-cmake/host-config.cmake.in @@ -0,0 +1,54 @@ +############################################################################### +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC +# and CHAI project contributors. See the CHAI LICENSE file for details. +# +# SPDX-License-Identifier: (BSD-3-Clause) +############################################################################### + +# Config related to compiler +set(CMAKE_C_COMPILER "@CMAKE_C_COMPILER@" CACHE PATH "") +set(CMAKE_CXX_COMPILER "@CMAKE_CXX_COMPILER@" CACHE PATH "") +set(CMAKE_Fortran_COMPILER "@CMAKE_Fortran_COMPILER@" CACHE PATH "") +set(CMAKE_CXX_STANDARD "@CMAKE_CXX_STANDARD@" CACHE STRING "") + +set(CMAKE_C_FLAGS "@CMAKE_C_FLAGS@" CACHE STRING "") +set(CMAKE_CXX_FLAGS "@CMAKE_CXX_FLAGS@" CACHE STRING "") +set(CMAKE_Fortran_FLAGS "@CMAKE_Fortran_FLAGS@" CACHE STRING "") +set(CMAKE_EXE_LINKER_FLAGS "@CMAKE_EXE_LINKER_FLAGS@" CACHE STRING "") + +set(CMAKE_EXPORT_COMPILE_COMMANDS TRUE CACHE BOOL "") + +set(BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE "@BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE@" CACHE STRING "") + +# MPI +set(ENABLE_MPI @ENABLE_MPI@ CACHE BOOL "") +set(MPI_C_COMPILER "@MPI_C_COMPILER@" CACHE PATH "") +set(MPI_CXX_COMPILER "@MPI_CXX_COMPILER@" CACHE PATH "") +set(MPI_Fortran_COMPILER "@MPI_Fortran_COMPILER@" CACHE PATH "") +set(MPIEXEC_EXECUTABLE "@MPIEXEC_EXECUTABLE@" CACHE PATH "") +set(MPIEXEC_NUMPROC_FLAG "@MPIEXEC_NUMPROC_FLAG@" CACHE STRING "") + +# CUDA +set(ENABLE_CUDA @ENABLE_CUDA@ CACHE BOOL "") + +if(ENABLE_CUDA) + set(CUDA_TOOLKIT_ROOT_DIR "@CUDA_TOOLKIT_ROOT_DIR@" CACHE PATH "") + set(CMAKE_CUDA_COMPILER "@CMAKE_CUDA_COMPILER@" CACHE PATH "") + set(CMAKE_CUDA_STANDARD "@CMAKE_CUDA_STANDARD@" CACHE STRING "") + set(CMAKE_CUDA_HOST_COMPILER "@CMAKE_CUDA_HOST_COMPILER@" CACHE PATH "") + set(CMAKE_CUDA_SEPARABLE_COMPILATION @CMAKE_CUDA_SEPARABLE_COMPILATION@ CACHE BOOL "") + set(CMAKE_CUDA_ARCHITECTURES "@CMAKE_CUDA_ARCHITECTURES@" CACHE STRING "") + set(CMAKE_CUDA_FLAGS "@CMAKE_CUDA_FLAGS@" CACHE STRING "") +endif() + +# HIP +set(ENABLE_HIP @ENABLE_HIP@ CACHE BOOL "") + +if(ENABLE_HIP) + set(HIP_ROOT_DIR "@HIP_ROOT_DIR@" CACHE PATH "") + set(HIP_CLANG_PATH "@HIP_CLANG_PATH@" CACHE PATH "") + set(CMAKE_HIP_ARCHITECTURES "@CMAKE_HIP_ARCHITECTURES@" CACHE STRING "") + set(CMAKE_EXE_LINKER_FLAGS "@CMAKE_EXE_LINKER_FLAGS@" CACHE STRING "") +endif() + +set(CHAI_DIR "@CMAKE_INSTALL_PREFIX@" CACHE PATH "") diff --git a/tests/install/using-with-cmake/using-with-cmake.cpp b/tests/install/using-with-cmake/using-with-cmake.cpp new file mode 100644 index 00000000..6effe7f9 --- /dev/null +++ b/tests/install/using-with-cmake/using-with-cmake.cpp @@ -0,0 +1,21 @@ +//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC +// and CHAI project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: (BSD-3-Clause) +//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~// + +#include "chai/ManagedArray.hpp" +#include + +int main(int, char**) +{ + constexpr std::size_t N{1024}; + chai::ManagedArray a(N); + + for (std::size_t i = 0; i < N; i++) { + a[i] = i; + } + + a.free(); +} diff --git a/tests/integration/CMakeLists.txt b/tests/integration/CMakeLists.txt index b422be2b..c8338b0c 100644 --- a/tests/integration/CMakeLists.txt +++ b/tests/integration/CMakeLists.txt @@ -1,15 +1,15 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## set (chai_integration_test_depends chai umpire gtest) -blt_list_append(TO chai_integration_test_depends ELEMENTS cuda IF ${ENABLE_CUDA}) -blt_list_append(TO chai_integration_test_depends ELEMENTS hip IF ${ENABLE_HIP}) -blt_list_append(TO chai_integration_test_depends ELEMENTS openmp IF ${ENABLE_OPENMP}) +blt_list_append(TO chai_integration_test_depends ELEMENTS cuda IF ${CHAI_ENABLE_CUDA}) +blt_list_append(TO chai_integration_test_depends ELEMENTS blt::hip IF ${CHAI_ENABLE_HIP}) +blt_list_append(TO chai_integration_test_depends ELEMENTS openmp IF ${CHAI_ENABLE_OPENMP}) # ManagedArray tests blt_add_executable( @@ -39,14 +39,14 @@ if (CHAI_ENABLE_MANAGED_PTR) NAME managed_ptr_test COMMAND managed_ptr_tests) endif () - -if (ENABLE_RAJA_PLUGIN) + +if (CHAI_ENABLE_RAJA_PLUGIN) set(raja_test_depends ${chai_integration_test_depends} RAJA) blt_add_executable( - NAME raja-chai-tests + NAME raja-chai-tests SOURCES raja-chai-tests.cpp DEPENDS_ON ${raja_test_depends}) @@ -58,9 +58,10 @@ if (ENABLE_RAJA_PLUGIN) raja-chai-tests PUBLIC ${PROJECT_BINARY_DIR}/include) - if (ENABLE_RAJA_NESTED_TEST) + if (CHAI_ENABLE_RAJA_NESTED_TEST) +#raja kernel tests blt_add_executable( - NAME raja-chai-nested-tests + NAME raja-chai-nested-tests SOURCES raja-chai-nested.cpp DEPENDS_ON ${raja_test_depends}) @@ -71,5 +72,21 @@ if (ENABLE_RAJA_PLUGIN) target_include_directories( raja-chai-nested-tests PUBLIC ${PROJECT_BINARY_DIR}/include) + +#raja launch tests + blt_add_executable( + NAME raja-chai-launch-tests + SOURCES raja-chai-launch.cpp + DEPENDS_ON ${raja_test_depends}) + + blt_add_test( + NAME raja-chai-launch-tests + COMMAND raja-chai-launch-tests) + + target_include_directories( + raja-chai-launch-tests + PUBLIC ${PROJECT_BINARY_DIR}/include) + endif () + endif () diff --git a/tests/integration/managed_array_tests.cpp b/tests/integration/managed_array_tests.cpp index 32a03217..28f4e11b 100644 --- a/tests/integration/managed_array_tests.cpp +++ b/tests/integration/managed_array_tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/integration/managed_ptr_tests.cpp b/tests/integration/managed_ptr_tests.cpp index 55a51e9d..579dbe35 100644 --- a/tests/integration/managed_ptr_tests.cpp +++ b/tests/integration/managed_ptr_tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/integration/raja-chai-launch.cpp b/tests/integration/raja-chai-launch.cpp new file mode 100644 index 00000000..9ece1e75 --- /dev/null +++ b/tests/integration/raja-chai-launch.cpp @@ -0,0 +1,409 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. +// +// SPDX-License-Identifier: BSD-3-Clause +////////////////////////////////////////////////////////////////////////////// +/// +/// Source file containing tests for CHAI in RAJA nested loops. +/// +/// +#include +#include +#include + +#include +#include +#include + +#include "RAJA/RAJA.hpp" + +using namespace RAJA; +using namespace std; + +#include "chai/ArrayManager.hpp" +#include "chai/ManagedArrayView.hpp" +#include "chai/ManagedArray.hpp" + +#include "gtest/gtest.h" + +// TODO: add hip policy for these tests. +#if defined(RAJA_ENABLE_CUDA) +#define PARALLEL_RAJA_DEVICE __device__ +#elif defined(RAJA_ENABLE_OPENMP) +#define PARALLEL_RAJA_DEVICE +#else +#define PARALLEL_RAJA_DEVICE +#endif + +#define CUDA_TEST(X, Y) \ + static void cuda_test_##X##_##Y(); \ + TEST(X, Y) { cuda_test_##X##_##Y(); } \ + static void cuda_test_##X##_##Y() + +/* + * Simple tests using nested::forall and View + */ +CUDA_TEST(Chai, LaunchSimple) +{ + + using LAUNCH_POLICY = RAJA::LaunchPolicy; + using LOOP_POLICY_0 = RAJA::LoopPolicy; + using LOOP_POLICY_1 = RAJA::LoopPolicy; + +#if defined(RAJA_ENABLE_CUDA) + + const bool async = false; + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy>; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + +#elif defined(RAJA_ENABLE_OPENMP) + + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + +#else + using PARALLEL_LAUNCH_POLICY = LAUNCH_POLICY; + using PARALLEL_LOOP_POLICY_0 = LOOP_POLICY_0; + using PARALLEL_LOOP_POLICY_1 = LOOP_POLICY_1; +#endif + + const int X = 16; + const int Y = 16; + + chai::ManagedArray v1(X * Y); + chai::ManagedArray v2(X * Y); + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + int index = j * X + i; + v1[index] = index; + }); + }); + }); + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + int index = j * X + i; + v2[index] = v1[index] * 2.0f; + }); + }); + }); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + int index = j * X + i; + ASSERT_FLOAT_EQ(v1[index], index * 1.0f); + ASSERT_FLOAT_EQ(v2[index], index * 2.0f); + }); + }); + }); + + v1.free(); + v2.free(); +} + +CUDA_TEST(Chai, LaunchView) +{ + + using LAUNCH_POLICY = RAJA::LaunchPolicy; + using LOOP_POLICY_0 = RAJA::LoopPolicy; + using LOOP_POLICY_1 = RAJA::LoopPolicy; + +#if defined(RAJA_ENABLE_CUDA) + + const bool async = false; + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy>; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + +#elif defined(RAJA_ENABLE_OPENMP) + + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + +#else + + using PARALLEL_LAUNCH_POLICY = LAUNCH_POLICY; + using PARALLEL_LOOP_POLICY_0 = LOOP_POLICY_0; + using PARALLEL_LOOP_POLICY_1 = LOOP_POLICY_1; +#endif + + const int X = 16; + const int Y = 16; + + chai::ManagedArray v1_array(X * Y); + chai::ManagedArray v2_array(X * Y); + + using view = chai::ManagedArrayView>; + + view v1(v1_array, X, Y); + view v2(v2_array, X, Y); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + v1(i, j) = (i + (j * X)) * 1.0f; + }); + }); + }); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + v2(i, j) = v1(i, j) * 2.0f; + }); + }); + }); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + ASSERT_FLOAT_EQ(v2(i, j), v1(i, j) * 2.0f); + }); + }); + }); + + v1_array.free(); + v2_array.free(); +} + +CUDA_TEST(Chai, LaunchMultiView) +{ + + using LAUNCH_POLICY = RAJA::LaunchPolicy; + using LOOP_POLICY_0 = RAJA::LoopPolicy; + using LOOP_POLICY_1 = RAJA::LoopPolicy; + +#if defined(RAJA_ENABLE_CUDA) + + const bool async = false; + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy>; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + +#elif defined(RAJA_ENABLE_OPENMP) + + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; +#else + + using PARALLEL_LAUNCH_POLICY = LAUNCH_POLICY; + using PARALLEL_LOOP_POLICY_0 = LOOP_POLICY_0; + using PARALLEL_LOOP_POLICY_1 = LOOP_POLICY_1; +#endif + + const int X = 16; + const int Y = 16; + + chai::ManagedArray v1_array(X * Y); + chai::ManagedArray v2_array(X * Y); + + chai::ManagedArray all_arrays[2]; + all_arrays[0] = v1_array; + all_arrays[1] = v2_array; + + // default MultiView + using view = chai::ManagedArrayMultiView>; + view mview(all_arrays, RAJA::Layout<2>(X, Y)); + + // MultiView with index in 1st position + using view1p = chai::ManagedArrayMultiView, 1>; + view1p mview1p(all_arrays, RAJA::Layout<2>(X, Y)); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + mview(0, i, j) = (i + (j * X)) * 1.0f; + }); + }); + }); + + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + // use both MultiViews + mview(1, i, j) = mview1p(i, 0, j) * 2.0f; + }); + }); + }); + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::RangeSegment(0, X), [&] (int i) { + RAJA::loop(ctx, RAJA::RangeSegment(0, Y), [&] (int j) { + ASSERT_FLOAT_EQ(mview(1, i, j), mview(0, i, j) * 2.0f); + }); + }); + }); + + + v1_array.free(); + v2_array.free(); +} + +/////////////////////////////////////////////////////////////////////////// +// +// Example LTimes kernel test routines +// +// Demonstrates a 4-nested loop, the use of complex nested policies and +// the use of strongly-typed indices +// +// This routine computes phi(m, g, z) = SUM_d { ell(m, d)*psi(d,g,z) } +// +/////////////////////////////////////////////////////////////////////////// + +RAJA_INDEX_VALUE_T(IM, int, "IM"); +RAJA_INDEX_VALUE_T(ID, int, "ID"); +RAJA_INDEX_VALUE_T(IG, int, "IG"); +RAJA_INDEX_VALUE_T(IZ, int, "IZ"); + +void runLTimesTests(Index_type num_moments, + Index_type num_directions, + Index_type num_groups, + Index_type num_zones) +{ + // allocate data + // phi is initialized to all zeros, the others are randomized + chai::ManagedArray L_data(num_moments * num_directions); + chai::ManagedArray psi_data(num_directions * num_groups * num_zones); + chai::ManagedArray phi_data(num_moments * num_groups * num_zones); + + RAJA::forall( + RAJA::RangeSegment(0, (num_moments * num_directions)), + [=](int i) { + L_data[i] = i+2; + }); + + RAJA::forall( + RAJA::RangeSegment(0, (num_directions * num_groups * num_zones)), + [=](int i) { psi_data[i] = 2*i+1; }); + + RAJA::forall( + RAJA::RangeSegment(0, (num_moments * num_groups * num_zones)), + [=](int i) { phi_data[i] = 0.0; }); + + using LView = chai::TypedManagedArrayView, IM, ID>; + + // psi(d, g, z) : 2 -> z is stride-1 dimension + using PsiView = chai::TypedManagedArrayView, ID, IG, IZ>; + + // phi(m, g, z) : 2 -> z is stride-1 dimension + using PhiView = chai::TypedManagedArrayView, IM, IG, IZ>; + + std::array L_perm {{0, 1}}; + LView L(L_data, + RAJA::make_permuted_layout({{num_moments, num_directions}}, L_perm)); + + std::array psi_perm {{0, 1, 2}}; + PsiView psi(psi_data, + RAJA::make_permuted_layout({{num_directions, num_groups, num_zones}}, psi_perm)); + + std::array phi_perm {{0, 1, 2}}; + PhiView phi(phi_data, + RAJA::make_permuted_layout({{num_moments, num_groups, num_zones}}, phi_perm)); + +#if defined(RAJA_ENABLE_CUDA) + const bool async = false; + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy>; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_2 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_3 = RAJA::LoopPolicy; + +#elif defined(RAJA_ENABLE_OPENMP) + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_2 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_3 = RAJA::LoopPolicy; +#else + using PARALLEL_LAUNCH_POLICY = RAJA::LaunchPolicy; + using PARALLEL_LOOP_POLICY_0 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_1 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_2 = RAJA::LoopPolicy; + using PARALLEL_LOOP_POLICY_3 = RAJA::LoopPolicy; +#endif + + RAJA::launch + (RAJA::LaunchParams(RAJA::Teams(1), RAJA::Threads(10)), + [=] RAJA_HOST_DEVICE(RAJA::LaunchContext ctx) + { + RAJA::loop(ctx, RAJA::TypedRangeSegment(0, num_moments), [&] (IM m) { + RAJA::loop(ctx, RAJA::TypedRangeSegment(0, num_groups), [&] (IG g) { + RAJA::loop(ctx, RAJA::TypedRangeSegment(0, num_zones ), [&] (IZ z) { + RAJA::loop(ctx, RAJA::TypedRangeSegment(0, num_directions ), [&] (ID d) { + phi(m, g, z) += L(m, d) * psi(d, g, z); + }); + }); + }); + }); + + }); + + RAJA::forall( + RAJA::TypedRangeSegment(0, num_moments), [=] (IM m) { + for (IG g(0); g < num_groups; ++g) { + for (IZ z(0); z < num_zones; ++z) { + double total = 0.0; + for (ID d(0); d < num_directions; ++d) { + double val = L(m, d) * psi(d, g, z); + total += val; + } + ASSERT_FLOAT_EQ(total, phi(m, g, z)); + } + } + }); + + L_data.free(); + psi_data.free(); + phi_data.free(); +} + +TEST(Chai, LaunchLTimes) +{ + // runLTimesTests(2, 0, 7, 3); + runLTimesTests(2, 3, 7, 3); + runLTimesTests(2, 3, 32, 4); + runLTimesTests(25, 96, 8, 32); + runLTimesTests(100, 15, 7, 13); +} diff --git a/tests/integration/raja-chai-nested.cpp b/tests/integration/raja-chai-nested.cpp index 152a18a2..edffd064 100644 --- a/tests/integration/raja-chai-nested.cpp +++ b/tests/integration/raja-chai-nested.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/integration/raja-chai-tests.cpp b/tests/integration/raja-chai-tests.cpp index f43340e6..6cabbe91 100644 --- a/tests/integration/raja-chai-tests.cpp +++ b/tests/integration/raja-chai-tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/unit/CMakeLists.txt b/tests/unit/CMakeLists.txt index d2236989..c453bc67 100644 --- a/tests/unit/CMakeLists.txt +++ b/tests/unit/CMakeLists.txt @@ -1,14 +1,14 @@ ############################################################################## -# Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -# project contributors. See the COPYRIGHT file for details. +# Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +# project contributors. See the CHAI LICENSE file for details. # # SPDX-License-Identifier: BSD-3-Clause ############################################################################## set (chai_unit_test_depends chai umpire gtest) -blt_list_append(TO chai_unit_test_depends ELEMENTS cuda IF ${ENABLE_CUDA}) -blt_list_append(TO chai_unit_test_depends ELEMENTS hip IF ${ENABLE_HIP}) +blt_list_append(TO chai_unit_test_depends ELEMENTS cuda IF ${CHAI_ENABLE_CUDA}) +blt_list_append(TO chai_unit_test_depends ELEMENTS blt::hip IF ${CHAI_ENABLE_HIP}) blt_add_executable( NAME managed_array_unit_tests diff --git a/tests/unit/array_manager_unit_tests.cpp b/tests/unit/array_manager_unit_tests.cpp index 3ea44ddc..abc4197a 100644 --- a/tests/unit/array_manager_unit_tests.cpp +++ b/tests/unit/array_manager_unit_tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/unit/managed_array_unit_tests.cpp b/tests/unit/managed_array_unit_tests.cpp index 0d3e3dae..628096f0 100644 --- a/tests/unit/managed_array_unit_tests.cpp +++ b/tests/unit/managed_array_unit_tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause ////////////////////////////////////////////////////////////////////////////// diff --git a/tests/unit/managed_ptr_unit_tests.cpp b/tests/unit/managed_ptr_unit_tests.cpp index f3cfe4ae..a1ab6c63 100644 --- a/tests/unit/managed_ptr_unit_tests.cpp +++ b/tests/unit/managed_ptr_unit_tests.cpp @@ -1,6 +1,6 @@ ////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-20, Lawrence Livermore National Security, LLC and CHAI -// project contributors. See the COPYRIGHT file for details. +// Copyright (c) 2016-24, Lawrence Livermore National Security, LLC and CHAI +// project contributors. See the CHAI LICENSE file for details. // // SPDX-License-Identifier: BSD-3-Clause //////////////////////////////////////////////////////////////////////////////