NVIDIA GPU Driver Install on RHEL 9 and Ubuntu 24.04
NVIDIA GPU Driver Install on RHEL 9.5
The following procedure is for installing the NVIDIA GPU drivers on RHEL 9.5. Note that this is for a single-node install. Scale with nodeshell and arguments to make these steps unattended as needed.
Register and subscribe a RHEL system to the Red Hat Customer Portal using Red Hat Subscription-Manager
This is required because some of the packages required are only available for registered systems.
subscription-manager register --username <username> --password <password>
subscription-manager release --set=9.5
subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms
dnf install -y wget
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
rpm -ivh epel-release-latest-9.noarch.rpm
dnf install dnf-plugin-config-manager
crb enable
Prerequisites
DOCA must be installed before installing the GPU drivers to make sure the nvidia-peermem kernel module gets the right Infiniband symbols.
Enable update repos
subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
Install newer kernel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel kernel-core kernel-modules-core kernel-modules
Install newer kernel devel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel-devel kernel-devel-matched kernel-headers kernel-modules-extra
Update kernel tools and kernel abi stablelists
Note that this still will update the existing kernel tools packages, as multiple versions of these kernel tools on the system as the same time.
dnf install kernel-tools kernel-tools-libs kernel-abi-stablelists
Install dkms
dnf install dkms
Disable update repos
subscription-manager repos --disable=rhel-9-for-x86_64-appstream-rpms
subscription-manager repos --disable=rhel-9-for-x86_64-baseos-rpms
Reboot the system using the newly installed kernel
reboot
Install GPU driver local repo
rpm -ivh <path to nvidia local repo RPM>/nvidia-driver-local-repo-rhel9-570.124.06-1.0-1.x86_64.rpm
Clean DNF/YUM cache
dnf clean all
Backup grub.cfg from ESP
When installing the nvidia-kmod-common package from the NVIDIA 570.124.06 local driver repo on RHEL 9.5 on an EFI-booted system, the /boot/efi/EFI/redhat/grub.cfg file, which is a “stub” grub.cfg in the EFI system partition (ESP) that only points to /boot/grub2 for the full grub.cfg, is overwritten with a full grub.cfg and the grub.cfg in /boot/grub2 is not updated. To work around this, backup the grub.cfg in the ESP (this will be restored and the grub.cfg file in /boot/grub2 fixed after installing the GPU drivers):
cp /boot/efi/EFI/redhat/grub.cfg ~/grub.cfg.PRE-NVIDIA
Install GPU drivers
Note The nvidia-fabric-manager package is only necesssary on 8-GPU HGX configs.
dnf install nvidia-driver-cuda kmod-nvidia-open-dkms nvidia-fabric-manager
Restore backed up stub grub.cfg and fix grub.cfg in /boot/grub2
cp -f /boot/efi/EFI/redhat/grub.cfg /boot/grub2
cp -f ~/grub.cfg.PRE-NVIDIA /boot/efi/EFI/redhat95/grub.cfg
Reboot
reboot
Check driver status
The following should show the correct driver version installed for the correct kernel version:
dkms status
The nvidia driver should be loaded and the nouveau driver should not be loaded:
lsmod | grep -i nvidia
lsmod | grep -i nouveau
Make sure the nouveau driver never loaded during the boot process:
dmesg | grep -i nouveau
Make sure the version of the nvidia driver running is correct:
cat /sys/module/nvidia/version
Start the nvidia-persistenced service and make sure its running OK:
systemctl start nvidia-persistenced
systemctl status nvidia-persistenced
Start the nvidia-fabricmanager service and make sure its running OK (note, this is only necessary for and should only be run on 8-GPU HGX systems only):
systemctl start nvidia-fabricmanager
systemctl status nvidia-fabricmanager
Check that nvidia-smi reports all expected GPUs:
nvidia-smi
Check the NVLINKs (where applicable–this applies to all HGX systems and systems with PCIe GPUs with NVLINK bridge cards installed). All expected links should show up at expected bandwidth:
nvidia-smi nvlink -s
Check the NVLINK fabric status (applicable for HGX 8-way configurations)
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
The output from that command should appear as follows:
GPU Fabric GUID : 0x7215545ecf79e88f
Inforom Version
Image Version : G525.0225.00.05
--
Fabric
State : Completed
Status : Success
NVIDIA GPU Driver Install on Ubuntu 24.04.2
The following procedure is for installing the NVIDIA GPU drivers on Ubuntu 24.04.2.
NOTE This is for a single-node install. Scale with nodeshell. Arguments to make these steps unattended have been included.
Install the NVIDIA driver local repo
Install the NVIDIA driver local repo package. Note that while this can be redundant with the CUDA local repo package, since the CUDA local repo package also contains the NVIDIA drivers, there can be cases where a different version of the NVIDIA driver from that contained in the version of CUDA that is being used is desired. Because of that, this procedure details installing the NVIDIA GPU drivers from the NVIDIA driver local repo.
dpkg -i /cluster/drivers/nvidia/570.124.06/nvidia-driver-local-repo-ubuntu2404-570.124.06_1.0-1_amd64.deb
Enroll the NVIDIA driver local repo GPG key
Enroll the GPG key for the NVIDIA driver local repo:
cp /var/nvidia-driver-local-repo-ubuntu2404-570.124.06/nvidia-driver-local-D67F55A1-keyring.gpg /usr/share/keyrings/
Pin the NVIDIA GPU driver version to that from the NVIDIA GPU driver local repo
The Ubuntu 24.04 OS repos contain many versions of the NVIDIA GPU drivers. To make sure the version installed is the one from the NVIDIA GPU driver local repo installed above, pin the version as follows (note that in this case the 570.124.06 NVIDIA GPU driver version is being used):
cat << EOF > /etc/apt/preferences.d/nvidia
Package: src:*nvidia*:any src:cuda-drivers:any src:cuda-compat:any
Pin: version 570.124.06-0ubuntu1
Pin-Priority: 1000
EOF
Update the apt repo cache
Update the apt repo cache for the newly installed NVIDIA driver local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Install the NVIDIA GPU driver packages
Install the NVIDIA GPU driver packages. The following examples will install the minimal number of packages necessary, suitable for a light-weight compute node install. Note that this step should be done after DOCA is installed, otherwise the nvidia-peermem module will not be compiled with the correct InfiniBand support and may fail to load.
In most cases the open drivers should be installed, as follows:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y libnvidia-compute-570 nvidia-dkms-570-open nvidia-compute-utils-570 nvidia-utils-570 libnvidia-cfg1-570
However, in some cases issues have been observed with the open drivers (e.g., when using 5th Generation AMD EPYC(TM) Processors). In this case the proprietary drivers should be installed instead:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y libnvidia-compute-570 nvidia-dkms-570 nvidia-compute-utils-570 nvidia-utils-570 libnvidia-cfg1-570
Extract the contents of the CUDA local repo package
The CUDA repo is required to install the GPU Direct Storage (GDS) kernel module and utility packages. While the CUDA local repo package can simply be installed locally on each compute node, this can take up a lot of local storage space. A more efficient way of making the CUDA local repo available to all compute nodes is to extract the files from the CUDA local repo package to a network share available to all compute nodes. In the following example, the /cluster directory is the network share mounted on all compute notes, the CUDA local repo package has been placed in the /cluster/software/cuda/12.8.1 directory, and the /cluster/software/cuda/12.8.1/ubuntu-repo directory is the location where the CUDA local repo contents will reside.
cd /cluster/software/cuda/12.8.1
mkdir ubuntu-repo
dpkg-deb -x cuda-repo-ubuntu2404-12-8-local_12.8.1-570.124.06-1_amd64.deb ubuntu-repo
Enroll the CUDA local repo GPG key
Enroll the GPG key for the CUDA local repo:
cp /cluster/software/cuda/12.8.1/ubuntu-repo/var/cuda-repo-ubuntu2404-12-8-local/cuda-B2775641-keyring.gpg /usr/share/keyrings/
Prioritize the CUDA local repo:
To prioritize the CUDA local repo, run the following (note the cuda-ubuntu-2404.pin file has been placed in the /cluster/software/cuda/12.8.1/ directory in this case):
cp /cluster/software/cuda/12.8.1/cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
Copy the CUDA repo doc files locally
Installing the CUDA local repo, aside from setting up the repo itself, does install the docs for the repo. To replicate this in the case that the CUDA local repo package wasn’t actually installed, the doc files can be copied into place. The following command will do that, using the example directories:
cp -a /cluster/software/cuda/12.8.1/ubuntu-repo/usr/share/doc/cuda-repo-ubuntu2404-12-8-local /usr/share/doc
Setup apt repo list file for CUDA local repo
Since the CUDA local repo package wasn’t actually installed, an apt repo list file has to be created to enable the CUDA local repo package contents extracted onto the network share in the step above. The following commands will set that up for the example path:
cat << EOF > /etc/apt/sources.list.d/cuda-ubuntu2404-12-8-local.list
deb [signed-by=/usr/share/keyrings/cuda-B2775641-keyring.gpg] file:///cluster/software/cuda/12.8.1/ubuntu-repo/var/cuda-repo-ubuntu2404-12-8-local /
EOF
Update the apt repo cache
Update the apt repo cache for the newly installed CUDA local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Install the GDS kernel module package
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fs
Install the GDS utility packages
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-gds
OPTIONAL Install the nvidia-fabric manager packages
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fabricmanager-570 libnvidia-nscq-570 nvidia-imex-570
OPTIONAL Install additional nvidia-fabric manager packages for B200
DEBIAN_FRONTEND=noninteractive apt-get install -V -y /cluster/drivers/nvidia/570.124.06/nvlsm_2025.01.5-1_amd64.deb
Reboot
reboot
Check driver status
The following should show the correct driver version installed for the correct kernel version:
dkms status
The nvidia driver should be loaded and the nouveau driver should not be loaded:
lsmod | grep -i nvidia
lsmod | grep -i nouveau
Make sure the nouveau driver never loaded during the boot process:
dmesg | grep -i nouveau
Make sure the version of the nvidia driver running is correct:
cat /sys/module/nvidia/version
Start the nvidia-persistenced service and make sure its running OK:
systemctl start nvidia-persistenced
systemctl status nvidia-persistenced
Start the nvidia-fabricmanager service and make sure its running OK (note, this is only necessary for and should only be run on 8-GPU HGX systems only):
systemctl start nvidia-fabricmanager
systemctl status nvidia-fabricmanager
Check that nvidia-smi reports all expected GPUs:
nvidia-smi
Check the NVLINKs (where applicable–this applies to all HGX systems and systems with PCIe GPUs with NVLINK bridge cards installed). All expected links should show up at expected bandwidth:
nvidia-smi nvlink -s
Check the NVLINK fabric status (applicable for HGX 8-way configurations)
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
The output from that command should appear as follows:
GPU Fabric GUID : 0x7215545ecf79e88f
Inforom Version
Image Version : G525.0225.00.05
--
Fabric
State : Completed
Status : Success
NVIDIA GPU Driver Install on RHEL 9.6
The following procedure is for installing the NVIDIA GPU drivers on RHEL 9.6. Note that this is for a single-node install. Scale with nodeshell and arguments to make these steps unattended as needed.
Register and subscribe a RHEL system to the Red Hat Customer Portal using Red Hat Subscription-Manager
This is required because some of the packages required are only available for registered systems.
subscription-manager register --username <username> --password <password>
subscription-manager release --set=9.6
subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms
dnf install -y wget
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
rpm -ivh epel-release-latest-9.noarch.rpm
dnf install dnf-plugin-config-manager
crb enable
Prerequisites
DOCA must be installed before installing the GPU drivers to make sure the nvidia-peermem kernel module gets the right Infiniband symbols.
Enable update repos
subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
Install newer kernel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel kernel-core kernel-modules-core kernel-modules
Install newer kernel devel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel-devel kernel-devel-matched kernel-headers kernel-modules-extra
Update kernel tools and kernel abi stablelists
Note that this still will update the existing kernel tools packages, as multiple versions of these kernel tools on the system as the same time.
dnf install kernel-tools kernel-tools-libs kernel-abi-stablelists
Install dkms
dnf install dkms
Disable update repos
subscription-manager repos --disable=rhel-9-for-x86_64-appstream-rpms
subscription-manager repos --disable=rhel-9-for-x86_64-baseos-rpms
Reboot the system using the newly installed kernel
reboot
Install GPU driver local repo
rpm -ivh <path to nvidia local repo RPM>/nvidia-driver-local-repo-rhel9-580.65.06-1.0-1.x86_64.rpm
Clean DNF/YUM cache
dnf clean all
Install GPU drivers
dnf install nvidia-driver-cuda kmod-nvidia-open-dkms
OPTIONAL Install nvidia-fabric manager packages
Note The nvidia-fabric-manager package is only necesssary on 8-GPU HGX configs.
dnf install nvidia-fabric-manager libnvidia-nscq libnvsdm nvidia-imex
OPTIONAL Install nvidia-fabric manager packages for B200
Note The nvidia-fabric-manager packages for B200 are only necesssary on B200 8-GPU HGX configs.
dnf install nvidia-fabric-manager libnvidia-nscq libnvsdm nvidia-imex collectx-bringup mft-autocomplete mft-oem nvlsm
Extract the contents of the CUDA local repo package
The CUDA repo is required to install the GPU Direct Storage (GDS) kernel module and utility packages.
rpm -ivh /cluster/software/cuda/13.0.0/cuda-repo-rhel9-13-0-local-13.0.0_580.65.06-1.x86_64.rpm
Clean DNF/YUM cache
dnf clean all
Install the GDS kernel module package
dnf install nvidia-fs
Install the GDS utility packages
dnf install nvidia-gds
Reboot
reboot
Check driver status
The following should show the correct driver version installed for the correct kernel version:
dkms status
The nvidia driver should be loaded and the nouveau driver should not be loaded:
lsmod | grep -i nvidia
lsmod | grep -i nouveau
Make sure the nouveau driver never loaded during the boot process:
dmesg | grep -i nouveau
Make sure the version of the nvidia driver running is correct:
cat /sys/module/nvidia/version
Start the nvidia-persistenced service and make sure its running OK:
systemctl start nvidia-persistenced
systemctl status nvidia-persistenced
Start the nvidia-fabricmanager service and make sure its running OK (note, this is only necessary for and should only be run on 8-GPU HGX systems only):
systemctl start nvidia-fabricmanager
systemctl status nvidia-fabricmanager
Check that nvidia-smi reports all expected GPUs:
nvidia-smi
Check the NVLINKs (where applicable–this applies to all HGX systems and systems with PCIe GPUs with NVLINK bridge cards installed). All expected links should show up at expected bandwidth:
nvidia-smi nvlink -s
Check the NVLINK fabric status (applicable for HGX 8-way configurations)
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
The output from that command should appear as follows:
GPU Fabric GUID : 0x7215545ecf79e88f
Inforom Version
Image Version : G525.0225.00.05
--
Fabric
State : Completed
Status : Success
For best GPU direct and GPU storage direct performance, IOMMU needs to be disabled or set to pass-through and PCIe Access Control Services (ACS) need to be disabled.
Since for large core count systems (i.e., >= 256 cores) IOMMU can’t be disabled or interrupt handling can be impacted, it is recommended to set IOMMU to pass-through. One way to do this is to include “intel_iommu=on iommu=pt” or “amd_iommu=on iommu=pt” on the kernel command line (depending on if the system has Intel or AMD CPUs). This method is recommended since if IOMMU, even if disabled from the UEFI settings perspective, if it is enabled from a Linux kernel perspective, then Linux kernel can also enable ACS. Adding this in a persistent way to the kernel command line can be achieved by modifying the “GRUB_CMDLINE_LINUX_DEFAULT” entry in the /etc/default/grub file. For example, change:
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200"
to
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200 intel_iommu=on iommu=pt"
in /etc/default/grub and then run
grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
NOTE It is recommended also to back up the /boot/grub2/grub.cfg file before running the grub2-mkconfig command.
Disabling ACS can on some systems be done using UEFI settings, since the OS can enable this despite the value of the UEFI setting, it is recommended to disable ACS using the following script:
#!/bin/bash
for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do
# skip if it doesn't support ACS
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
if [ $? -ne 0 ]; then
continue
fi
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done
This script should be set to run on boot.
Checking the iommu status can be checked with
cat /proc/cmdline
and checking the ACS status can be checked by running
/usr/local/cuda/gds/tools/gdscheck -p
NVIDIA GPU Driver Install on Ubuntu 24.04.3
The following procedure is for installing the NVIDIA GPU drivers on Ubuntu 24.04.3.
NOTE This is for a single-node install. Scale with nodeshell. Arguments to make these steps unattended have been included.
Install the NVIDIA driver local repo
Install the NVIDIA driver local repo package. Note that while this can be redundant with the CUDA local repo package, since the CUDA local repo package also contains the NVIDIA drivers, there can be cases where a different version of the NVIDIA driver from that contained in the version of CUDA that is being used is desired. Because of that, this procedure details installing the NVIDIA GPU drivers from the NVIDIA driver local repo.
dpkg -i /cluster/drivers/nvidia/580.65.06/nvidia-driver-local-repo-ubuntu2404-580.65.06_1.0-1_amd64.deb
Enroll the NVIDIA driver local repo GPG key
Enroll the GPG key for the NVIDIA driver local repo:
cp /var/nvidia-driver-local-repo-ubuntu2404-580.65.06/nvidia-driver-local-C9531D52-keyring.gpg /usr/share/keyrings/
Pin the NVIDIA GPU driver version to that from the NVIDIA GPU driver local repo
The Ubuntu 24.04 OS repos contain many versions of the NVIDIA GPU drivers. To make sure the version installed is the one from the NVIDIA GPU driver local repo installed above, pin the version as follows (note that in this case the 580.65.06 NVIDIA GPU driver version is being used):
cat << EOF > /etc/apt/preferences.d/nvidia
Package: src:*nvidia*:any src:cuda-drivers:any src:cuda-compat:any
Pin: version 580.65.06-0ubuntu1
Pin-Priority: 1000
EOF
Update the apt repo cache
Update the apt repo cache for the newly installed NVIDIA driver local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Install the NVIDIA GPU driver packages
Install the NVIDIA GPU driver packages. The following examples will install the minimal number of packages necessary, suitable for a light-weight compute node install. Note that this step should be done after DOCA is installed, otherwise the nvidia-peermem module will not be compiled with the correct InfiniBand support and may fail to load.
In most cases the open drivers should be installed, as follows:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y libnvidia-compute-580 nvidia-dkms-580-open nvidia-compute-utils-580 nvidia-utils-580 libnvidia-cfg1-580
However, in some cases issues have been observed with the open drivers (e.g., when using 5th Generation AMD EPYC(TM) Processors). In this case the proprietary drivers should be installed instead:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y libnvidia-compute-580 nvidia-dkms-580 nvidia-compute-utils-580 nvidia-utils-580 libnvidia-cfg1-580
Install the CUDA local repo package
The CUDA repo is required to install the GPU Direct Storage (GDS) kernel module and utility packages.
dpkg -i /cluster/software/cuda/13.0.0/cuda-repo-ubuntu2404-13-0-local_13.0.0-580.65.06-1_amd64.deb
Enroll the CUDA local repo GPG key
Enroll the GPG key for the CUDA local repo:
cp /var/cuda-repo-ubuntu2404-13-0-local/cuda-BFD0EC79-keyring.gpg /usr/share/keyrings/
Prioritize the CUDA local repo:
To prioritize the CUDA local repo, run the following (note the cuda-ubuntu-2404.pin file has been placed in the /cluster/software/cuda/13.0.0/ directory in this case):
cp /cluster/software/cuda/13.0.0/cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
Update the apt repo cache
Update the apt repo cache for the newly installed CUDA local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Install the GDS kernel module package
yes | DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fs
Install the GDS utility packages
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-gds
OPTIONAL Install the nvidia-fabric manager packages (needed for HGX 8 GPU configs)
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fabricmanager-580 libnvidia-nscq-580 nvidia-imex-580
OPTIONAL Install additional nvidia-fabric manager packages for B200 (needed for HGX 8 GPU B200 configs)
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvlsm
Reboot
reboot
Check driver status
The following should show the correct driver version installed for the correct kernel version:
dkms status
The nvidia driver should be loaded and the nouveau driver should not be loaded:
lsmod | grep -i nvidia
lsmod | grep -i nouveau
Make sure the nouveau driver never loaded during the boot process:
dmesg | grep -i nouveau
Make sure the version of the nvidia driver running is correct:
cat /sys/module/nvidia/version
Start the nvidia-persistenced service and make sure its running OK:
systemctl start nvidia-persistenced
systemctl status nvidia-persistenced
Start the nvidia-fabricmanager service and make sure its running OK (note, this is only necessary for and should only be run on 8-GPU HGX systems only):
systemctl start nvidia-fabricmanager
systemctl status nvidia-fabricmanager
Check that nvidia-smi reports all expected GPUs:
nvidia-smi
Check the NVLINKs (where applicable–this applies to all HGX systems and systems with PCIe GPUs with NVLINK bridge cards installed). All expected links should show up at expected bandwidth:
nvidia-smi nvlink -s
Check the NVLINK fabric status (applicable for HGX 8-way configurations)
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
The output from that command should appear as follows:
GPU Fabric GUID : 0x7215545ecf79e88f
Inforom Version
Image Version : G525.0225.00.05
--
Fabric
State : Completed
Status : Success
For best GPU direct and GPU storage direct performance, IOMMU needs to be disabled or set to pass-through and PCIe Access Control Services (ACS) need to be disabled.
Since for large core count systems (i.e., >= 256 cores) IOMMU can’t be disabled or interrupt handling can be impacted, it is recommended to set IOMMU to pass-through. One way to do this is to include “intel_iommu=on iommu=pt” or “amd_iommu=on iommu=pt” on the kernel command line (depending on if the system has Intel or AMD CPUs). This method is recommended since if IOMMU, even if disabled from the UEFI settings perspective, if it is enabled from a Linux kernel perspective, then Linux kernel can also enable ACS. Adding this in a persistent way to the kernel command line can be achieved by modifying the “GRUB_CMDLINE_LINUX_DEFAULT” entry in the /etc/default/grub file. For example, change:
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200"
to
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200 intel_iommu=on iommu=pt"
in /etc/default/grub and then run
update-grub2
NOTE It is recommended also to back up the /boot/grub/grub.cfg file before running the update-grub2 command.
Disabling ACS can on some systems be done using UEFI settings, since the OS can enable this despite the value of the UEFI setting, it is recommended to disable ACS using the following script:
#!/bin/bash
for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do
# skip if it doesn't support ACS
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
if [ $? -ne 0 ]; then
continue
fi
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done
This script should be set to run on boot.
Checking the iommu status can be checked with
cat /proc/cmdline
and checking the ACS status can be checked by running
/usr/local/cuda/gds/tools/gdscheck -p
NVIDIA GPU Driver v580.126.09 Install on RHEL 9.6
The following procedure is for installing the NVIDIA GPU drivers on RHEL 9.6. Note that this is for a single-node install. Scale with nodeshell and arguments to make these steps unattended as needed.
Register and subscribe a RHEL system to the Red Hat Customer Portal using Red Hat Subscription-Manager
This is required because some of the packages required are only available for registered systems.
subscription-manager register --username <username> --password <password>
subscription-manager release --set=9.6
subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms
dnf install -y wget
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
rpm -ivh epel-release-latest-9.noarch.rpm
dnf install dnf-plugin-config-manager
crb enable
Prerequisites
DOCA must be installed before installing the GPU drivers to make sure the nvidia-peermem kernel module gets the right Infiniband symbols.
Enable update repos
subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
Install newer kernel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel kernel-core kernel-modules-core kernel-modules
Install newer kernel devel packages
Note that these steps install a new version as opposed to replacing the older kernel version(s), so the system can be booted to the older kernel version(s) if needed.
dnf install kernel-devel kernel-devel-matched kernel-headers kernel-modules-extra
Update kernel tools and kernel abi stablelists
Note that this still will update the existing kernel tools packages, as multiple versions of these kernel tools on the system as the same time.
dnf install kernel-tools kernel-tools-libs kernel-abi-stablelists
Install dkms
dnf install dkms
Disable update repos
subscription-manager repos --disable=rhel-9-for-x86_64-appstream-rpms
subscription-manager repos --disable=rhel-9-for-x86_64-baseos-rpms
Reboot the system using the newly installed kernel
reboot
Install GPU driver local repo
rpm -ivh <path to nvidia local repo RPM>/nvidia-driver-local-repo-rhel9-580.126.09-1.0-1.x86_64.rpm
Clean DNF/YUM cache
dnf clean all
dnf makecache
Install GPU drivers
dnf module enable nvidia-driver:open-dkms/fm
dnf install nvidia-driver-cuda kmod-nvidia-open-dkms
OPTIONAL Install nvidia-fabric manager packages
Note The nvidia-fabric-manager package is only necesssary on 8-GPU HGX configs.
dnf install nvidia-fabric-manager libnvidia-nscq libnvsdm nvidia-imex
OPTIONAL Install nvidia-fabric manager packages for B200
Note The nvidia-fabric-manager packages for B200 are only necesssary on B200 8-GPU HGX configs.
dnf install nvidia-fabric-manager libnvidia-nscq libnvsdm nvidia-imex collectx-bringup mft-autocomplete mft-oem nvlsm
Extract the contents of the CUDA local repo package
The CUDA repo is required to install the GPU Direct Storage (GDS) kernel module and utility packages.
rpm -ivh /cluster/software/cuda/13.0.2/cuda-repo-rhel9-13-0-local-13.0.2_580.95.05-1.x86_64.rpm
Clean DNF/YUM cache
dnf clean all
dnf makecache
Install the GDS kernel module package
dnf install nvidia-fs
Install the GDS utility packages
dnf install nvidia-gds
Reboot
reboot
Check driver status
The following should show the correct driver version installed for the correct kernel version:
dkms status
The nvidia driver should be loaded and the nouveau driver should not be loaded:
lsmod | grep -i nvidia
lsmod | grep -i nouveau
Make sure the nouveau driver never loaded during the boot process:
dmesg | grep -i nouveau
Make sure the version of the nvidia driver running is correct:
cat /sys/module/nvidia/version
Start the nvidia-persistenced service and make sure its running OK:
systemctl start nvidia-persistenced
systemctl status nvidia-persistenced
Start the nvidia-fabricmanager service and make sure its running OK (note, this is only necessary for and should only be run on 8-GPU HGX systems only):
systemctl start nvidia-fabricmanager
systemctl status nvidia-fabricmanager
Check that nvidia-smi reports all expected GPUs:
nvidia-smi
Check the NVLINKs (where applicable–this applies to all HGX systems and systems with PCIe GPUs with NVLINK bridge cards installed). All expected links should show up at expected bandwidth:
nvidia-smi nvlink -s
Check the NVLINK fabric status (applicable for HGX 8-way configurations)
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
The output from that command should appear as follows:
GPU Fabric GUID : 0x7215545ecf79e88f
Inforom Version
Image Version : G525.0225.00.05
--
Fabric
State : Completed
Status : Success
For best GPU direct and GPU storage direct performance, IOMMU needs to be disabled or set to pass-through and PCIe Access Control Services (ACS) need to be disabled.
Since for large core count systems (i.e., >= 256 cores) IOMMU can’t be disabled or interrupt handling can be impacted, it is recommended to set IOMMU to pass-through. One way to do this is to include “intel_iommu=on iommu=pt” or “amd_iommu=on iommu=pt” on the kernel command line (depending on if the system has Intel or AMD CPUs). This method is recommended since if IOMMU, even if disabled from the UEFI settings perspective, if it is enabled from a Linux kernel perspective, then Linux kernel can also enable ACS. Adding this in a persistent way to the kernel command line can be achieved by modifying the “GRUB_CMDLINE_LINUX_DEFAULT” entry in the /etc/default/grub file. For example, change:
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200"
to
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200 intel_iommu=on iommu=pt"
in /etc/default/grub and then run
grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
NOTE It is recommended also to back up the /boot/grub2/grub.cfg file before running the grub2-mkconfig command.
Disabling ACS can on some systems be done using UEFI settings, since the OS can enable this despite the value of the UEFI setting, it is recommended to disable ACS using the following script:
#!/bin/bash
for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do
# skip if it doesn't support ACS
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
if [ $? -ne 0 ]; then
continue
fi
sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done
This script should be set to run on boot.
Checking the iommu status can be checked with
cat /proc/cmdline
and checking the ACS status can be checked by running
/usr/local/cuda/gds/tools/gdscheck -p
NVIDIA GPU Driver and CUDA Toolkit Install on Ubuntu 24.04.4
The following procedure is for installing the NVIDIA GPU drivers and CUDA toolkit on Ubuntu 24.04.4.
NOTE This is for a single-node install. Scale with nodeshell. Arguments to make these steps unattended have been included.
NVIDIA GPU Driver
Install the NVIDIA driver local repo
Install the NVIDIA driver local repo package. Note that while this can be redundant with the CUDA local repo package, since the CUDA local repo package also contains the NVIDIA drivers, there can be cases where a different version of the NVIDIA driver from that contained in the version of CUDA that is being used is desired. Because of that, this procedure details installing the NVIDIA GPU drivers from the NVIDIA driver local repo.
dpkg -i /cluster/drivers/nvidia/595.71.05/nvidia-driver-local-repo-ubuntu2404-595.71.05_1.0-1_amd64.deb
Enroll the NVIDIA driver local repo GPG key
Enroll the GPG key for the NVIDIA driver local repo:
cp /var/nvidia-driver-local-repo-ubuntu2404-595.71.05/nvidia-driver-local-0EC8F1BA-keyring.gpg /usr/share/keyrings/
Update the apt repo cache
Update the apt repo cache for the newly installed NVIDIA driver local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Pin the NVIDIA GPU driver version to that from the NVIDIA GPU driver local repo
The Ubuntu 24.04 OS repos contain many versions of the NVIDIA GPU drivers. To make sure the version installed is the one from the NVIDIA GPU driver local repo installed above, pin the version as follows (note that in this case the 595.71.05 NVIDIA GPU driver version is being used):
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-driver-pinning-595.71.05
Install the NVIDIA GPU driver packages
Install the NVIDIA GPU driver packages. The following examples will install the minimal number of packages necessary, suitable for a light-weight compute node install. Note that this step should be done after DOCA is installed, otherwise the nvidia-peermem module will not be compiled with the correct InfiniBand support and may fail to load.
As of Ubuntu 24.04.4 and the 595.71.05 NVIDIA GPU driver version, no issues have been seen with the open drivers, so only the open driver install case is shown:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y libnvidia-compute nvidia-dkms-open
Setup nvidia_peermem dkms dependency
The following sets up a dkms dependency so the nvidia_peermem module will be automatically rebuilt if DOCA is updated:
echo 'BUILD_DEPENDS[4]="mlnx-ofed-kernel"' >> /etc/dkms/nvidia.conf
echo 'BUILD_DEPENDS_REBUILD=yes' >> /etc/dkms/nvidia.conf
OPTIONAL Install the nvidia-fabric manager packages (needed for HGX 8 GPU configs)
lspci | grep -i nvswitch && DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fabricmanager libnvidia-nscq nvidia-imex
OPTIONAL Install additional nvidia-fabric manager packages for B200 (needed for HGX 8 GPU B200 configs)
lspci | grep -i mell | cut -f 1 -d ' ' | xargs -i /bin/bash -c "lspci -vvn -s {}" | grep -i umbriel > /dev/null && DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-fabricmanager libnvidia-nscq nvidia-imex libnvsdm nvlsm
Setup nvidia_peermem to load on boot
echo "nvidia-peermem" >> /etc/modules-load.d/nvidia-peermem.conf
Update the initramfs
Update the initramfs to make sure all of the new kernel modules in particular are loaded on boot:
update-initramfs -u
Reboot
reboot
Check driver status
The following script will check the following:
- the nvidia driver is setup through dkms
- the expected nvidia modules are loaded (including nvidia_peermem)
- that the nouveau driver is not loaded
- that the nouveau driver has never loaded for this boot
- that the nvidia driver version is as expected
- that the nvidia-persistenced service is running
- that the nvidia-fabricmanager service is running (where applicable)
- the NVIDIA GPUs seen by nvidia-smi
- the NVLink point-to-point status seen by nvidia-smi
- the NVLink fabric status seen by nvidia-smi
#!/bin/bash
# Define text coloring for clean output
RED='\033[0;31m'
RED2=$'\e[0;31m'
GREEN='\033[0;32m'
GREEN2=$'\e[0;32m'
NC='\033[0m' # No Color
NC2=$'\e[0m' # No Color
NVIDIA_DRIVER_VERSION="595.71.05"
NVIDIA_FS_DRIVER_VERSION="2.28.4"
dkms status | grep 'nvidia/'${NVIDIA_DRIVER_VERSION} | grep -q "installed" && \
printf "%-40s %s\n" "dkms nvidia driver status:" ${GREEN2}installed${NC2} || \
printf "%-40s %s\n" "dkms nvidia driver status:" ${RED2}FAIL${NC2}
### Move to cuda check since nvidia-fs not installed until cuda installed
#dkms status | grep 'nvidia-fs/'${NVIDIA_FS_DRIVER_VERSION} | grep -q "installed" && \
# printf "%-40s %s\n" "dkms nvidia-fs driver status:" ${GREEN2}installed${NC2} || \
# printf "%-40s %s\n" "dkms nvidia-fs driver status:" ${RED2}FAIL${NC2}
#
echo
### Move nvidia-fs check to cuda check since nvidia-fs not installed until cuda installed
#for i in nvidia nvidia_fs nvidia_peermem nvidia_modeset nvidia_drm nvidia_uvm; do
for i in nvidia nvidia_peermem nvidia_modeset nvidia_drm nvidia_uvm; do
lsmod | grep -q '^'${i} && \
printf "%-40s %s\n" "${i} driver loaded:" ${GREEN2}loaded${NC2} || \
printf "%-40s %s\n" "${i} driver loaded:" ${RED2}FAIL${NC2}
done
echo
lsmod | grep -q '^nouveau' && \
printf "%-40s %s\n" "nouveau driver not currently loaded:" ${RED2}loaded${NC2} || \
printf "%-40s %s\n" "nouveau driver not currently loaded:" ${GREEN2}"not loaded"${NC2}
echo
dmesg | grep -q 'nouveau' && \
printf "%-40s %s\n" "nouveau driver never loaded this boot:" ${RED2}loaded${NC2} || \
printf "%-40s %s\n" "nouveau driver never loaded this boot:" ${GREEN2}"never loaded"${NC2}
echo
[ ! -d "/sys/module/nvidia" ] && LNDV="NONE" || LNDV=$(cat /sys/module/nvidia/version)
echo ${LNDV} | grep -q ${NVIDIA_DRIVER_VERSION} && \
printf "%-40s %s\n" "loaded nvidia driver version:" ${GREEN2}${LNDV}${NC2} || \
printf "%-40s %s\n" "loaded nvidia driver version:" ${RED2}${LNDV}${NC2}
echo
systemctl is-active nvidia-persistenced | grep -q '^active' && \
printf "%-40s %s\n" "nvidia-persistenced active:" ${GREEN2}$(systemctl is-active nvidia-persistenced)${NC2} || \
printf "%-40s %s\n" "nvidia-persistenced active:" ${RED2}$(systemctl is-active nvidia-persistenced)${NC2}
systemctl is-enabled nvidia-persistenced | grep -q '^enabled' && \
printf "%-40s %s\n" "nvidia-persistenced enabled:" ${GREEN2}$(systemctl is-enabled nvidia-persistenced)${NC2} || \
printf "%-40s %s\n" "nvidia-persistenced enabled:" ${RED2}$(systemctl is-enabled nvidia-persistenced)${NC2}
echo
## For Pre-Blackwell HGX 8
if lspci | grep -qi nvswitch; then
systemctl is-active nvidia-fabricmanager | grep -q '^active' && \
printf "%-40s %s\n" "nvidia-fabricmanager active:" ${GREEN2}$(systemctl is-active nvidia-fabricmanager)${NC2} || \
printf "%-40s %s\n" "nvidia-fabricmanager active:" ${RED2}$(systemctl is-active nvidia-fabricmanager)${NC2}
systemctl is-enabled nvidia-fabricmanager | grep -q '^enabled' && \
printf "%-40s %s\n" "nvidia-fabricmanager enabled:" ${GREEN2}$(systemctl is-enabled nvidia-fabricmanager)${NC2} || \
printf "%-40s %s\n" "nvidia-fabricmanager enabled:" ${RED2}$(systemctl is-enabled nvidia-fabricmanager)${NC2}
## For Blackwell (NVLink5)
elif lspci | grep -i mell | cut -f 1 -d ' ' | xargs -i /bin/bash -c "lspci -vvn -s {}" | grep -i umbriel > /dev/null; then
systemctl is-active nvidia-fabricmanager | grep -q '^active' && \
printf "%-40s %s\n" "nvidia-fabricmanager active:" ${GREEN2}$(systemctl is-active nvidia-fabricmanager)${NC2} || \
printf "%-40s %s\n" "nvidia-fabricmanager active:" ${RED2}$(systemctl is-active nvidia-fabricmanager)${NC2}
systemctl is-enabled nvidia-fabricmanager | grep -q '^enabled' && \
printf "%-40s %s\n" "nvidia-fabricmanager enabled:" ${GREEN2}$(systemctl is-enabled nvidia-fabricmanager)${NC2} || \
printf "%-40s %s\n" "nvidia-fabricmanager enabled:" ${RED2}$(systemctl is-enabled nvidia-fabricmanager)${NC2}
else
echo -e "\e[1mnvidia-fabric manager not applicable for this system\e[0m"
fi
echo
echo -e "\e[1mnvidia-smi -L output:\e[0m"
echo -e "\e[1m=====================\e[0m"
[ -f "/usr/bin/nvidia-smi" ] && \
nvidia-smi -L ||
echo -e "${RED}nvidia-smi not found${NC}"
echo
echo -e "\e[1mnvidia-smi nvlink -s output:\e[0m"
echo -e "\e[1m============================\e[0m"
[ -f "/usr/bin/nvidia-smi" ] && \
nvidia-smi nvlink -s ||
echo -e "${RED}nvidia-smi not found${NC}"
echo
echo -e "\e[1mnvidia-smi -q -i 0 | grep -i -A 2 Fabric -s output:\e[0m"
echo -e "\e[1m===================================================\e[0m"
[ -f "/usr/bin/nvidia-smi" ] && \
nvidia-smi -q -i 0 | grep -i -A 2 Fabric ||
echo -e "${RED}nvidia-smi not found${NC}"
exit
CUDA Toolkit
Prioritize the CUDA local repo
To prioritize the CUDA local repo, run the following (note the cuda-ubuntu-2404.pin file has been placed in the /cluster/software/cuda/13.2.1/ directory in this case):
cp /cluster/software/cuda/13.2.1/cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
Install the CUDA local repo package
The CUDA repo is required to install the GPU Direct Storage (GDS) kernel module and utility packages.
dpkg -i /cluster/software/cuda/13.2.1/cuda-repo-ubuntu2404-13-2-local_13.2.1-595.58.03-1_amd64.deb
Enroll the CUDA local repo GPG key
Enroll the GPG key for the CUDA local repo:
cp /var/cuda-repo-ubuntu2404-13-2-local/cuda-50F9113D-keyring.gpg /usr/share/keyrings/
Update the apt repo cache
Update the apt repo cache for the newly installed CUDA local repo:
DEBIAN_FRONTEND=noninteractive apt-get update -y
Install the CUDA runtime, compiler and library metapackages
The CUDA toolkit metapackage has dependencies on GUI packages–here the CUDA runtime, compiler and library metapackages are installed instead to avoid installing unnecessary GUI packages on compute nodes running headlessly:
DEBIAN_FRONTEND=noninteractive apt-get install -V -y cuda-runtime-13-2 cuda-compiler-13-2 cuda-libraries-dev-13-2
Install the GDS utility packages
Note, this will install the nvidia-fs kernel module as well.
DEBIAN_FRONTEND=noninteractive apt-get install -V -y nvidia-gds
For best GPU direct and GPU storage direct performance, IOMMU needs to be disabled or set to pass-through and PCIe Access Control Services (ACS) need to be disabled
Since for large core count systems (i.e., >= 256 cores) IOMMU can’t be disabled or interrupt handling can be impacted, it is recommended to set IOMMU to pass-through. One way to do this is to include “intel_iommu=on iommu=pt” or “amd_iommu=on iommu=pt” on the kernel command line (depending on if the system has Intel or AMD CPUs). This method is recommended since if IOMMU, even if disabled from the UEFI settings perspective, if it is enabled from a Linux kernel perspective, then Linux kernel can also enable ACS. Adding this in a persistent way to the kernel command line can be achieved by modifying the “GRUB_CMDLINE_LINUX_DEFAULT” entry in the /etc/default/grub file. The following script will back up the existing /etc/default/grub file, perform the modification and update the grub.conf file:
#!/bin/bash
### Script to setup IOMMU pass-through for Ubuntu 24.04
# Define text coloring for clean output
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color
## Attempt to get raw Vendor ID from /proc/cpuinfo
if [ -f /proc/cpuinfo ]; then
VENDOR_ID=$(grep -m 1 'vendor_id' /proc/cpuinfo | awk -F: '{print $2}' | tr -d '[:space:]')
fi
## Fallback to lscpu if /proc/cpuinfo didn't yield a vendor string
if [ -z "$VENDOR_ID" ] && command -v lscpu >/dev/null 2>&1; then
VENDOR_ID=$(lscpu | grep -i 'Vendor ID:' | awk -F: '{print $2}' | tr -d '[:space:]')
fi
## Fallback to lscpu Model Name if Vendor ID is still empty (common on ARM/Raspberry Pi)
if [ -z "$VENDOR_ID" ] && command -v lscpu >/dev/null 2>&1; then
VENDOR_ID=$(lscpu | grep -i 'Model name:' | awk -F: '{print $2}' | tr -d '[:space:]')
fi
## Process the identified vendor ID
case "$VENDOR_ID" in
*GenuineIntel*)
echo -e "Result: ${GREEN}Intel Corporation${NC} ($VENDOR_ID)"
IOMMU_CMD="intel_iommu=on"
;;
*AuthenticAMD*)
echo -e "Result: ${GREEN}AMD (Advanced Micro Devices)${NC} ($VENDOR_ID)"
IOMMU_CMD="amd_iommu=on"
;;
*Apple*)
echo -e "Result: ${RED}Apple Silicon${NC} ($VENDOR_ID)"
exit 1
;;
*Ampere*|*ARM*|*Cortex*)
echo -e "Result: ${RED}ARM Architecture${NC} ($VENDOR_ID)"
exit 1
;;
"")
echo -e "${RED}Error: Could not detect CPU vendor.${NC}"
exit 1
;;
*)
echo -e "Result: ${RED}Unknown / Other Vendor${NC} ($VENDOR_ID)"
exit 1
;;
esac
GRUB_FILE="/etc/default/grub"
BACKUP_FILE="${GRUB_FILE}.bak.$(date +%Y%m%d%H%M%S)"
PARAM="iommu=pt"
## Check if any iommu settings are already set in GRUB_CMDLINE_LINUX_DEFAULT
if grep -qE '^GRUB_CMDLINE_LINUX_DEFAULT=".*iommu.*"' "$GRUB_FILE"; then
echo -e "${RED}IOMMU settings in place already--check manually.${NC}"
exit 1
else
## Back up original config
cp "$GRUB_FILE" "$BACKUP_FILE"
echo "grub file backup created at $BACKUP_FILE"
echo "Adding $IOMMU_CMD to GRUB_CMDLINE_LINUX_DEFAULT..."
## Append iommu=pt and clean up any double spaces created by the change
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"/GRUB_CMDLINE_LINUX_DEFAULT="\1 '"$IOMMU_CMD"' '"$PARAM"'"/g' "$GRUB_FILE"
sed -i 's/ / /g' "$GRUB_FILE"
## Update the bootloader
if command -v update-grub2 &> /dev/null; then
echo "Updating GRUB..."
update-grub2
## Add in later for RHEL support?
#elif command -v grub2-mkconfig &> /dev/null; then
# echo "Updating GRUB..."
# grub2-mkconfig -o /boot/grub2/grub.cfg
else
echo -e "${RED}Error: Could not find grub update utility. Restoring backup grub file. Please update manually.${NC}"
cp "$BACKUP_FILE" "$GRUB_FILE"
exit 1
fi
fi
Disabling ACS can on some systems be done using UEFI settings, since the OS can enable this despite the value of the UEFI setting, it is recommended to disable ACS using the following script which will setup a service that disables ACS every boot:
#!/bin/bash
cat > /etc/systemd/system/disable-acs.service <<'EOF'
[Unit]
Description=Disable ACS
Requires=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/disable-acs.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
cat > /usr/local/bin/disable-acs.sh <<'EOF'
#!/bin/bash
# must be root to access extended PCI config space
if [ "$EUID" -ne 0 ]; then
echo "ERROR: $0 must be run as root"
exit 1
fi
for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do
# skip if it doesn't support ACS
setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
if [ $? -ne 0 ]; then
#echo "${BDF} does not support ACS, skipping"
continue
fi
echo "Disabling ACS on `lspci -s ${BDF}`"
setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
if [ $? -ne 0 ]; then
echo "Error disabling ACS on ${BDF}"
continue
fi
NEW_VAL=`setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk '{print $NF}'`
if [ "${NEW_VAL}" != "0000" ]; then
echo "Failed to disable ACS on ${BDF}"
continue
fi
done
exit 0
EOF
chmod +x /usr/local/bin/disable-acs.sh
systemctl daemon-reload
systemctl enable disable-acs.service
exit 0
Update the initramfs
Update the initramfs to make sure all of the new kernel modules in particular are loaded on boot:
update-initramfs -u
Reboot
reboot
Check GPU Direct Storage, IOMMU pass-through and ACS disabled status
The following script can be used to make sure the following is setup correctly:
- expected version of nvidia_fs driver is setup through dkms
- the nvidia_fs driver is loaded
- that iommu=pt is setup on the kernel command line
- that the kernel reports the expected IOMMU mode
- that NVIDIA GDS is working and ACS is disabled
#!/bin/bash
# Define text coloring for clean output
RED='\033[0;31m'
RED2=$'\e[0;31m'
GREEN='\033[0;32m'
GREEN2=$'\e[0;32m'
NC='\033[0m' # No Color
NC2=$'\e[0m' # No Color
NVIDIA_FS_DRIVER_VERSION="2.28.4"
dkms status | grep 'nvidia-fs/'${NVIDIA_FS_DRIVER_VERSION} | grep -q "installed" && \
printf "%-40s %s\n" "dkms nvidia-fs driver status:" ${GREEN2}installed${NC2} || \
printf "%-40s %s\n" "dkms nvidia-fs driver status:" ${RED2}FAIL${NC2}
echo
lsmod | grep -q '^nvidia_fs' && \
printf "%-40s %s\n" "nvidia_fs driver loaded:" ${GREEN2}loaded${NC2} || \
printf "%-40s %s\n" "nvidia_fs driver loaded:" ${RED2}FAIL${NC2}
###
# Setup iommu=pt
cat /proc/cmdline
dmesg | grep -i 'iommu: Default domain type'
#disable acs
/usr/local/cuda-13.2/gds/tools/gdscheck.py -p