In this set of exercises, we will take two example applications in tandem to show the effect of setting proper affinity to CPU cores and GPUs. Using the hello_jobstep example, we will see how affinity settings could be verified programmatically. Using the ghost exchange example, we will examine how setting affinity properly can help improve performance. We will examine different scenarios incrementally starting from not setting any affinity at all, to setting both CPU core and GPU device affinities. Please note that the hello_jobstep application is built with OpenMP® support while the Ghost Exchange Ver1 example does not have OpenMP support.

Follow the instructions in the sections below for cloning and building the two examples.

Clone the hello_jobstep code and build it as shown below:

To build the Ghost Exchange code, we need ROCm to be loaded in your environment and the ROCM_PATH environment variable be set up with the path to your ROCm installation. We will also need a build of OpenMPI (say ver 5.0.3) that uses UCX (say ver 1.16.x) built with ROCm. The following instructions assume that ROCM_PATH is set up, and that OpenMPI is installed at /opt/ompi-5.0.3.

Ver1 of the Ghost Exchange example uses offloading to GPU using the HIP programming model and a managed memory model. Using the managed memory model, the memory buffers are still initially allocated on host, but the OS will manage page migration and data movement across the PCIe link between the host and device. For this to happen when the Ghost Exchange example is run, we need to set up an environment variable, HSA_XNACK=1.

To understand how to set up affinity, we need to know the topology of the system we have. Two commands help us here. lscpu informs us the CPU hardware thread (HWT) count and how they are configured into NUMA domains. rocm-smi --showtoponuma shows the NUMA configuration for the GPU devices on the system. As an example, here are the details from each command on the node we are using for this tutorial.

The output of lscpu below shows that hardware threads 0-23 and 96-119 belong to NUMA domain 0, 24-47 and 120-143 belong to NUMA domain 1, and so on.

And the output of rocm-smi --showtoponuma shows that GPU device 0 belongs to NUMA domain 0, GPU 1 belongs to NUMA domain 1, etc. With this knowledge, we believe that it would be best if a process using GPU 0 is also pinned to run on a hardware thread in the range 0-23,96-119. The proximity of CPU cores to GPUs is determined by their association with a common NUMA domain.

Let us examine the case where we run only 1 MPI process per GPU device. In each case shown below, we will run both examples using the same settings and observe the output from each one.

In this case, we do not set up any affinity, so we see that each process landed in some HWT. In this case, we see HWTs 001, 121, 144, and 072. We also see all GPUs were available to each process in the RT_GPU_ID list for each one, and there was no assigned GPU for each process because GPU_ID = N/A.

The output for the Ghost Exchange example shows the same thing, and we see that the elapsed time for this run without any affinity settings is 3.55 seconds.

Next, we will set up GPU affinity by setting assigning a unique GPU device for each process via the environment variable, ROCR_VISIBLE_DEVICES. We use a script at ~/git/HPCTrainingExamples/MPI-examples/GhostExchange/GhostExchange_ArrayAssign/HIP/set_gpu_device_mi300a.sh to achieve this. Now, we see that process 0 ran on GPU device 0 (see GPU_ID field), and so on.

We can observe the GPU affinity set up correctly in the Ghost Exchange example too. In this case, we see that the application ran almost 3x faster with an elapsed time of 1.2 seconds.

Now, we will attempt to set CPU affinity only.

For the hello_jobstep example that supports OpenMP, we will use OpenMP environment variables, OMP_PLACES and OMP_PROC_BIND to specify placement and binding for each process. Now, we see that when using OMP_PLACES=threads and OMP_PROC_BIND=close, processes 0, 1, 2, and 3 ran on HWTs 0, 24, 48, and 72 respectively. Please note that you may see different behavior on your system, and hence exploring the OpenMP man page for the various settings for these environment variables may become unavoidable.

In the Ghost Exchange case, we need to use a different pinning mechanism for CPU threads as it is not built with OpenMP support. We resort to using OpenMPI binding options as shown below. And we see that setting CPU affinity alone is not sufficient.

Let us set both CPU and GPU affinity now.

And for Ghost exchange, we use a combination of mpirun binding options and the GPU affinity script. This improves the performance back to what we saw when we pinned the GPU device only. This shows that GPU affinity is very important for this application's performance.

Oversubscription of a GPU with multiple processes requires a more complex script that helps set up affinity such that neighboring ranks are closely packed on GPU devices (i.e., ranks 0 and 1 on GPU 0, ranks 2 and 3 on GPU 1, etc.) and use different cores on the NUMA domain that is closest to the selected GPU. The script shown below works perfectly for the hello_jobstep example as using GOMP_CPU_AFFINITY environment variable for setting CPU affinity requires an OpenMP based application.

For Ghost Exchange, we will use a combination of OpenMPI binding options and the packing script to set up GPU affinity:

For hello_jobstep:

And for Ghost Exchange:

Hope this tutorial helped you learn some useful tricks to set affinity properly. Happy pinning!