-
Created by
Marco Verlato, last updated on Sept 27, 2022
7 minute read
- be sure to blacklist the nouveau driver in one of the /etc/modprobe.d/*.conf files. If not, create a /etc/modprobe.d/gpu-blacklist.conf file with the two lines below:
blacklist nouveau options nouveau modeset=0
and then execute:
$ dracut --force; reboot
- look at the model of GPU(s) with lspci command, e.g.:
[root@cld-np-gpu-03 ~]# lspci -nn | grep NVI ca:00.0 3D controller [0302]: NVIDIA Corporation GA100GL [A30 PCIe] [10de:20b7] (rev a1) [root@cld-dfa-gpu-04 ~]# lspci -nn | grep NVI 17:00.0 3D controller [0302]: NVIDIA Corporation GA107GL [A2 / A16] [10de:25b6] (rev a1) ca:00.0 3D controller [0302]: NVIDIA Corporation GA107GL [A2 / A16] [10de:25b6] (rev a1)
in the first server there is one A30 GPU card, in the second one two A2 GPU cards.
Look at the driver available for the given GPU card at https://www.nvidia.com/Download/index.aspx?lang=en-us. Once selected the driver version for the proper OS, you can follow the instructions there and install the rpm. If something goes wrong with the rpm, try with the .run script as described in https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html#runfile, e.g.:
# yum install gcc kernel-devel (if not already installed) BASE_URL=https://us.download.nvidia.com/tesla DRIVER_VERSION=515.65.01 curl -fSsl -O $BASE_URL/$DRIVER_VERSION/NVIDIA-Linux-x86_64-$DRIVER_VERSION.run sh ./NVIDIA-Linux-x86_64-$DRIVER_VERSION.run # if something goes wrong, try: sh ./NVIDIA-Linux-x86_64-$DRIVER_VERSION.run --add-this-kernel sh ./NVIDIA-Linux-x86_64-$DRIVER_VERSION-custom.run
- check if things work (no reboot should be needed), e.g.:
[root@cld-dfa-gpu-04 ~]# nvidia-smi Fri Sep 2 17:37:04 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A2 Off | 00000000:17:00.0 Off | 0 | | 0% 33C P0 19W / 60W | 0MiB / 15356MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA A2 Off | 00000000:CA:00.0 Off | 0 | | 0% 35C P0 18W / 60W | 0MiB / 15356MiB | 1% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
- Install a CUDA Toolkit version compatible with the selected NVIDIA driver (see https://developer.nvidia.com/cuda-downloads):
# if needed, yum install make freeglut-devel libX11-devel libXi-devel libXmu-devel mesa-libGLU-devel freeimage-devel
BASE_URL=https://developer.download.nvidia.com/compute/cuda
CUDA_VERSION=11.7.1
DRIVER_VERSION=515.65.01
curl -fSsl -O $BASE_URL/$CUDA_VERSION/local_installers/cuda_${CUDA_VERSION}_${DRIVER_VERSION}_linux.run
sh ./cuda_${CUDA_VERSION}_${DRIVER_VERSION}_linux.run
- check if everything works. e.g.:
[root@cld-dfa-gpu-04 ~]# /usr/local/cuda-11.7/extras/demo_suite/deviceQuery /usr/local/cuda-11.7/extras/demo_suite/deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 2 CUDA Capable device(s) Device 0: "NVIDIA A2" CUDA Driver Version / Runtime Version 11.7 / 11.7 CUDA Capability Major/Minor version number: 8.6 Total amount of global memory: 14967 MBytes (15693709312 bytes) (10) Multiprocessors, (128) CUDA Cores/MP: 1280 CUDA Cores GPU Max Clock rate: 1770 MHz (1.77 GHz) Memory Clock rate: 6251 Mhz Memory Bus Width: 128-bit L2 Cache Size: 2097152 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 1536 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 23 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > Device 1: "NVIDIA A2" CUDA Driver Version / Runtime Version 11.7 / 11.7 CUDA Capability Major/Minor version number: 8.6 Total amount of global memory: 14967 MBytes (15693709312 bytes) (10) Multiprocessors, (128) CUDA Cores/MP: 1280 CUDA Cores GPU Max Clock rate: 1770 MHz (1.77 GHz) Memory Clock rate: 6251 Mhz Memory Bus Width: 128-bit L2 Cache Size: 2097152 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 1536 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 202 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > > Peer access from NVIDIA A2 (GPU0) -> NVIDIA A2 (GPU1) : Yes > Peer access from NVIDIA A2 (GPU1) -> NVIDIA A2 (GPU0) : Yes deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.7, CUDA Runtime Version = 11.7, NumDevs = 2, Device0 = NVIDIA A2, Device1 = NVIDIA A2 Result = PASS