Install Prometheus

Reading instructions

• In the command line and configuration file examples in this guide, you can ignore all annotations that start with #.

Step1. Log in to the active management node, and then install Prometheus for the backup management node:

cd /root
source lico_env.local
ssh $backup_sms_ip dnf install -y prometheus-config

Step 2. Log in to the backup management node and copy the configuration file from the active management node:

cd /root
source lico_env.local
scp -r root@$sms_ip:/etc/prometheus/{cloud,local,rules,lico-prometheus.yml} /etc/prometheus/

Step 3. Edit the /etc/prometheus/lico-prometheus.yml file on the backup management node:

remote_write:
  - url: http://<$backup_sms_ip>:18101/receive
    write_relabel_configs:
      - source_labels:
          - __name__
        regex: '^(up|cpu_util|node_load1|node_load5|node_load15|node_memory_SReclaimable_bytes|node_memory_Buffers_bytes|node_memory_Cached_bytes|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_network_receive_bytes_total|node_network_transmit_bytes_total|node_network_receive_packets_total|node_network_transmit_packets_total|node_infiniband_port_data_received_bytes_total|node_infiniband_port_data_transmitted_bytes_total|node_filesystem_size_bytes|node_filesystem_avail_bytes|DCGM_FI_PROF_SM_ACTIVE|DCGM_FI_DEV_MIG_MAX_SLICES|DCGM_FI_DEV_FB_TOTAL|DCGM_FI_DEV_FB_RESERVED|DCGM_FI_DEV_FB_USED_PERCENT|DCGM_FI_DEV_FB_USED|DCGM_FI_DEV_FB_FREE|DCGM_FI_DEV_GPU_TEMP|DCGM_FI_DEV_GPU_UTIL|DCGM_FI_DEV_MEM_COPY_UTIL|DCGM_FI_DEV_MIG_MODE|DCGM_FI_DEV_PCIE_MAX_LINK_GEN|DCGM_FI_DEV_PCIE_LINK_GEN|DCGM_FI_DRIVER_VERSION|DCGM_FI_DEV_UUID|DCGM_FI_DEV_NAME|SL_JOB_PROCESS_CPU_UTIL|SL_JOB_PROCESS_MEMORY_USED|SL_JOB_PROCESS_GPU_UTIL|SL_JOB_PROCESS_GPU_MEMORY_USAGE|SERVER_POWER|SERVER_TEMPERATURE|SERVER_HEALTH_LEVEL|SERVER_ACTIVE_EVENT_LEVEL|HL_NODE_VNC_SESSION|HL_CPU_LOGICAL_CORE|HL_GPU_PROC_NUM)$'
        action: keep

Step 4. Start services on the backup management node:

systemctl enable prometheus --now