Limitations

Version 8.0

• When MiG is activated for a GPU, in the Home page there will be no information regarding the number of MiG instances present in the cluster. The number of in-use MiG instances is displayed correctly. On the Login page, there will be no monitoring data for the MiG instances.
• When MiG is activated for a GPU and a job is running on one of the MiG instances, the GPU column in the Process tab under Monitor → List View → Node details will not display correct information.
• When MiG is activated for a GPU and a job is running on one of the MiG instances, the GPU column in the Process tab under Jobs Monitoring will not display correct information.
• For the LLM Finetuning template there are the following limitations:
  • The job will fail if using the Default data type. The user must select INT8 or Pure BF16 as data types.
  • Finetuning will fail if using INT8 in multiple node mode.
• In Hybrid HPC → vGPU section, in the VM View tab, if you execute reboot and then immediately execute shutdown, the virtual machine status will be stuck in Shutting down state.

Version 8.1

• When the scheduler used in the cluster is either LSF or PBS in the Home page the number of used GPUs / MIG instances will be shown as 0.
• For the Node Manager there are the following limitations:
  • If the deployment fails and the user edit details about the node then only the Restart option should be used to restart the deployment. The Retry option won’t work.
  • The Node Manager supports the deployment of EL9 operating system, SLURM scheduler and NVIDIA GPU platforms.
• In Hybrid HPC → vGPU section, in the VM View tab, if you execute reboot and then immediately execute shutdown, the virtual machine status will be stuck in Shutting down state.
• If MIG is activated and the user choose to have multiple compute instances on the same GPU instance the monitoring of MIG instances will not display correct information.
• If MIG is activated in Monitor → Large Screen View there will be no data about the Utilization and Memory usage for the MIG instances.
• If MIG is activated in Monitor → Cluster View there will be no data displayed about MIG instances.
• Monitoring support for AMD GPUs requires the AMD Device Metrics Exporter to be installed by the user and configured manually into Prometheus for LiCO monitoring.
  • Please refer to the official AMD installation guide for detailed setup instructions: AMD Device Metrics Exporter – Singularity Installation
  • Note: If the exporter is not installed, AMD GPU metrics will not be available for monitoring.