Install SELF

The Spectral Element Library in Fortran can be built provided the following dependencies are met

• Fortran 2008 compliant compiler ( gfortran recommended )
• Cmake (v3.21 or greater)
• MPI, e.g. OpenMPI
• ROCm v5.7.0 or greater
• HDF5
• FluidNumerics/feq-parse
• (Optional) CUDA Toolkit, if you are building for Nvidia GPU hardware.

You can install SELF in two possible ways

1. Bare Metal Installation
2. Docker image

A bare metal installation will require that you have a CentOS/RHEL 7 or 8, SLES, Ubuntu 20.04 (focal) or 22.04 (jammy), or Windows operating system. You will also need to ensure that all of the dependencies are installed on your system before installing SELF.

A Docker image installation uses the Ubuntu 22.04 Docker image as a base and takes care of installing all of the dependencies for you. The resulting Docker image can be run using Docker or Singularity/Apptainer.

This documentation will walk through steps to install SELF using bare metal installation and the Docker image approaches.

Bare Metal Install

Install SELF with CMake

SELF comes with a CMake build system that defines the build and installation process. When you install SELF, you will install the following artifacts

• ${CMAKE_INSTALL_PREFIX}/lib/libself-static.a - A static library for the SELF API, in case you want to build your own programs and solvers using Spectral Element Methods.
• ${CMAKE_INSTALL_PREFIX}/lib/libself.so - A static library for the SELF API, in case you want to build your own programs and solvers using Spectral Element Methods.
• ${CMAKE_INSTALL_PREFIX}/include/*.mod - Module files generated by the Fortran compiler during the build process.
• ${CMAKE_INSTALL_PREFIX}/example/* - A set of example programs that run simple linear models (advection-diffusion) in 1-D, 2-D, or 3-D
• ${CMAKE_INSTALL_PREFIX}/test/* - A set of unit tests that exercise specific methods beneath the model classes.

This part of the documentation will provide you with an overview of the environment variables that control the build process, show you how to target different GPUs for GPU acceleration, and how to install SELF to a preferred directory on your system.

Build Variables

There are a number of environment variables you can use to control the behavior of the build and installation process. Importantly, some of these environment variables are necessary to tell the build system where dependencies can be found.

• CMAKE_HIP_ARCHITECTURES The target GPU architecture to build for (e.g. gfx906, gfx90a, sm_72)
• CMAKE_INSTALL_PREFIX The installation path for SELF
• CMAKE_BUILD_TYPE Type of build, one of Release, Debug, or Coverage

The default values of these variables will work for you if the following conditions are met * You have all of the necessary dependencies installed and visible in your PATH and LD_LIBRARY_PATH environment variables * ROCm is installed in /opt/rocm (the default location) * If you are targeting Nvidia GPUs, CUDA is installed in /usr/local/cuda (the default location)

Building for an Nvidia GPU

To build SELF for running on Nvidia GPUs, you will need to have both HIP and the CUDA toolkit installed on your system.

Next, you will need to set the CMAKE_HIP_ARCHITECTURES build variable to the microarchicture code for the specific GPU you are targeting (see the table below).

Install SELF

First, clone the SELF repository (if you haven't already)

git clone https://github.com/fluidnumerics/SELF ~/SELF

Next, create a build directory for Cmake to stage intermediate files

mkdir ~/SELF/build
cd ~/SELF/build

Use Cmake to build the make system

 FC=gfortran \
    cmake -DCMAKE_PREFIX_PATH=/opt/rocm
          -DCMAKE_HIP_ARCHITECTURES=gfx90a \
          -DCMAKE_INSTALL_PREFIX=${HOME}/opt/self \
          -DCMAKE_BUILD_TYPE=Release \
          ./ 

In this example, * FC=gfortran sets the fortran compiler to gfortran. * CMAKE_PREFIX_PATH instructs Cmake to search for Cmake configuration files for ROCm underneath /opt/rocm * CMAKE_HIP_ARCHITECTURES=gfx90a sets the target GPU architecture to the AMD MI200 series GPUs * CMAKE_INSTALL_PREFIX=${HOME}/opt/self sets the installation path for SELF to ${HOME}/opt/self; when setting this variable, make sure it is a location where you have read and write permissions * CMAKE_BUILD_TYPE=Release sets the build type to Release, which enables -O3 optimizations. If you are troubleshooting an issue, it is best to set this to Debug.

After running cmake, you can build SELF,

make VERBOSE=1

We recommend that you run the unit tests included with SELF,

ctest --test-dir ~/SELF/build/

If all of the tests pass, install SELF

make install

At the end of this process, the self application is installed under ${HOME}/opt/self/bin. Additionally, the SELF static library can be found under ${HOME}/opt/self/lib and the .mod files for all of the SELF modules are under ${HOME}/opt/self/include.

If you encounter any problems, feel free to open an new issue

Build a Docker Container

SELF comes with Docker files defined under the docker/ subdirectory. Currently, there are recipes for building container images for AMD GPUs (Dockerfile.rocm) and Nvidia GPUs (Dockerfile.cuda). The recipes will install all of SELF's dependencies and SELF in a container image based on the Ubuntu 22.04 image. To bake a SELF image, first clone the SELF repository and navigate to the source code directory

git clone https://github.com/fluidnumerics/SELF ${HOME}/SELF
cd ${HOME}/SELF

In this example, SELF is cloned to ${HOME}/SELF. From the main directory of the SELF repository, you can use docker build to build a container image.

docker build -f docker/rocm-5.7/amd/Dockerfile -t self:latest .

This command will create a Docker image for running SELF on AMD MI50 GPUs and the image is tagged self:latest; this is the name of the image that you will reference when running SELF.

By default, this will build SELF with double precision floating point arithmetic, no optimizations (debug build), and with GPU kernels offloaded to a AMD MI50 GPUs. You can customize the behavior of the build process by using build substitutions. The following build substitution variables are currently available

• _GPU_TARGET: GPU microarchitecture code to build for. Defaults to gfx906 (AMD MI50)