Upgrade from LiCO 7.2.0 to 7.2.1 for Container Deployment

Introduction

This Guide describes how to upgrade LiCO container from V7.2.0 to V7.2.1.

Prerequisites

The upgrade procedure supports LiCO V7.2.0 and higher versions. Upgrade from a lower version is not supported.
You have installed LiCO according to the official installation guide of the corresponding operating system.

Operating environment

Before upgrading LiCO, ensure the version of the operating system, OpenHPC, and Icinga are listed in the following table. To use LiCO 7.2.1 in the required operating system, install LiCO 7.2.1 by following the instructions of LiCO 7.2.1 Installation Guide (for EL8).

LiCO version	Red Hat (RHEL)	Rocky/EL8	Lenovo OpenHPC version	Icinga version
V7.2.0	8.6	8.6	2.6.2	2.13.8
V7.2.1	8.8	8.8	2.8	2.14.2

Prepare

1. Stop LiCO Container

lico-control stop
docker rename lico lico_7.2.0

2.Prepare to upgrade

Step 1. Log in to the management node.

Step 2. Edit /root/lico_release.local and update the environment variables listed in that file:

# Management node hostname
sms_name="head"
# IP address of management node
sms_ip="192.168.0.2"
# Old version of repository directory for LiCO and the version of LiCO in
# the current system
original_lico_repo_dir="/install/custom/lico-7.2.0"
# Old version of repository directory for LiCO-dep and
# the version of LiCO-dep in the current system
original_lico_dep_repo_dir="/install/custom/lico-dep-7.2.0"
# New version of repository directory for LiCO, please change it according to LiCO version
new_lico_repo_dir="/install/custom/lico-7.2.1"
# New version of LiCO repository directory for LiCO-dep,
# please change it according to LiCO-dep version
new_lico_dep_repo_dir="/install/custom/lico-dep-7.2.1"
# Link name of LiCO repository directory for LiCO
link_lico_repo_dir="/install/custom/lico"
# Link name of LiCO repository directory for LiCO-dep
link_lico_dep_repo_dir="/install/custom/lico-dep"
# Existing shared directories for LiCO
lico_share_dir="/home"
# Shared directories of repo file in LiCO Installation Guide
share_installer_dir="/install/installer"
# link name of directory for lico moniotr
link_lico_monitor_dir="/install/custom/lico-monitor"
# Local directory for for lico monitor, please change it according to this version.
lico_monitor_dir="/install/custom/lico-monitor-1.1.1"

Step 3. Save the changes to lico_release.local and reload the environment variables:

chmod 600 lico_release.local
source lico_release.local

3. Build LiCO image

Step 1. Prepare packages. Download the following packages and upload all of them to the /root directory:

Lenovo-OpenHPC-2.8.EL8.x86_64.tar: https://hpc.lenovo.com/lico/downloads/7.2.1/Lenovo-OpenHPC-2.8.EL8.x86_64.tar
lico-dep-7.2.1.el8.x86_64.tgz: https://hpc.lenovo.com/lico/downloads/7.2.1/lico-dep-7.2.1.el8.x86_64.tgz
authselect.tar.gz: https://hpc.lenovo.com/lico/downloads/7.2.1/examples/conf/authselect/authselect.tar.gz
LiCO 7.2.1 release package lico-release-7.2.1.el8.x86_64.tar.gz: https://commercial.lenovo.com/cn/en/signin
LiCO 7.2.1 docker package lico-docker-7.2.1.x86_64.tar.gz: https://commercial.lenovo.com/cn/en/signin
LiCO 7.2.1 monitor package openlico-monitor-1.1.1.x86_64.tgz: https://commercial.lenovo.com/cn/en/signin

Step 2. Prepare build environment:

cd /root
lico_container_workspace="/root/lico-workspace/7.2.1"
mkdir -p $lico_container_workspace
tar -xzvf lico-docker-7.2.1.x86_64.tar.gz -C $lico_container_workspace
mv Lenovo-OpenHPC-2.8.EL8.x86_64.tar $lico_container_workspace
mv lico-dep-7.2.1.el8.x86_64.tgz $lico_container_workspace
mv lico-release-7.2.1.el8.x86_64.tar.gz $lico_container_workspace
mv authselect.tar.gz $lico_container_workspace

cp $lico_container_workspace/lico-docker/Dockerfile \
$lico_container_workspace/lico-docker/scripts/build.sh $lico_container_workspace

\cp $lico_container_workspace/lico-docker/scripts/lico-control /usr/bin/

chmod +x $lico_container_workspace/build.sh
chmod +x /usr/bin/lico-control

Step 3. Build LiCO image:

cd $lico_container_workspace
./build.sh

4. Backup

backup_dir=/root/lico_7.2.0_backup/
mkdir -p $backup_dir
cd $lico_container_workspace/lico-docker/upgrade
python3 container-upgrade backup --backup-path $backup_dir --backup-db --backup-config --backup-sharedir

If the /opt/lico/cloud/azure/slurm.conf file exists, back it up:

cp /opt/lico/cloud/azure/slurm.conf /opt/lico/cloud/azure/slurm.conf.bak

Upgrade software on management node

Step 1. Upgrade LiCO icinga2 plugin:

systemctl stop icinga2
nodeshell all systemctl stop icinga2
rm -rf /opt/lico/pub/monitor
mkdir -p $lico_monitor_dir
cd /root
tar -xvzf openlico-monitor-1.1.1.x86_64.tgz -C $lico_monitor_dir
rm -rf $link_lico_monitor_dir
ln -s $lico_monitor_dir $link_lico_monitor_dir
dnf install -y python39
nodeshell all dnf install -y python39
cd $link_lico_monitor_dir
python3 $link_lico_monitor_dir/install.py -p python3.9

Step 2. Set and create a backup directory:

icinga_backup="/install/icinga2/backup"
mkdir -p $icinga_backup

Step 3. Configure icinga global template file on management node:

Backup the existing configuration files:

mv /etc/icinga2/zones.d/global-templates/commands.conf $icinga_backup

Download commands.conf from the following command and replace it to /etc/icinga2/zones.d/global-templates.

cd /etc/icinga2/zones.d/global-templates/
curl -O https://hpc.lenovo.com/lico/downloads/7.2.1/examples/conf/icinga2/commands/slurm/commands.conf

Modify the privileges

chown -R icinga:icinga /etc/icinga2/zones.d/global-templates

Step 4. Define icinga zone file:

Backup the existing configuration files:

cp /etc/icinga2/zones.d/master/hosts.conf $icinga_backup
mv /etc/icinga2/zones.d/master/service.conf $icinga_backup

Locate the configuration information of the Icinga 2 management node in the /etc/icinga2/zones.d/master/hosts.conf file, and set the configuration entry as vars.is_master = 1. Following is an example whose management node name is head.
```
object Host "head" {
   check_command = "hostalive"
   address = "192.168.0.1"
   vars.is_master = 1
   vars.agent_endpoint = name
}
```
Download service.conf from https://hpc.lenovo.com/lico/downloads/7.2.1/examples/conf/icinga2/ and upload it to /etc/icinga2/zones.d/master

Modify the privileges:

chown -R icinga:icinga /etc/icinga2/zones.d/master

Step 5. Enable influxdb writer feature for icinga:

Backup the existing configuration files:

cp /etc/icinga2/features-available/influxdb.conf $icinga_backup

Rebuild the icinga database in influxdb

influx -username <influxdb username> -password <influxdb password>
drop database icinga
create database icinga
use icinga
create retention policy "three_hour_only" on "icinga" duration 3h replication 1 default
exit

Install and configure Telegraf. For details, refer to the step “Deploy Telegraf to transfer the Icinga output and save the changes to Influxdb” of LiCO 7.2.1 Installation Guide (for EL8).
Download influxdb.conf from https://hpc.lenovo.com/lico/downloads/7.2.1/examples/conf/icinga2/ , upload influxdb.conf to /etc/icinga2/features-available/, and modify the influxdb writer configuration file referring to the Telegraf configuraiton.
```
host = "<telegraf server address>"
port = 9081
basic_auth = {
   username = "<telegraf authentication username>"
   password = "<telegraf authentication password>"
}
```

Step 6. Restart the icinga2 service to make the changes take effect.

nodeshell all systemctl restart icinga2
systemctl restart icinga2

Deploy LiCO 7.2 container

Step 1. Prepare the mount folder and configure the file

rm -rf /etc/lico
lico-control prepare

Step 2. merge the config file

cp $backup_dir/head/configure/lico/nodes.csv /etc/lico/
cd /root/lico-workspace/7.2.1/lico-docker/upgrade/
python3 container-upgrade merge-config --backup-path $backup_dir

If the /opt/lico/cloud/azure/slurm.conf.bak file exists, restore it:

mv /opt/lico/cloud/azure/slurm.conf.bak /opt/lico/cloud/azure/slurm.conf

Step 3. Edit the following configuration in the /etc/lico/lico.ini file

[DATABASE]
HOST = '127.0.0.1'
PORT = 3306

[TRANSITER]
HOST = '127.0.0.1'
PORT = 22

Step 4. Initial data for LiCO

lico-control init --mode  lico

lico-control init --mode  cloudtools

Step 5. Start the Container

lico-control run
lico-control docker-cmd remove lico_7.2.0

Rollback

Stop LiCO 7.2.1 container

lico-control stop
lico-control docker-cmd rename lico lico_7.2.1

cd $lico_container_workspace/lico-docker/upgrade/
python3 container-rollback --backup-path $backup_dir --restore-db --restore-sharedir

restore software in management node

systemctl stop icinga2
nodeshell all systemctl stop icinga2
rm -rf /opt/lico/pub/monitor
source lico_env.local 
mkdir -p $lico_monitor_dir
tar -xvzf openlico-monitor-1.1.0.x86_64.tgz -C $lico_monitor_dir
rm -rf $link_lico_monitor_dir
rm -rf $new_lico_monitor_dir
ln -s $lico_monitor_dir $link_lico_monitor_dir
python3 $link_lico_monitor_dir/install.py

cp $icinga_backup/commands.conf /etc/icinga2/zones.d/global-templates/commands.conf
cp $icinga_backup/hosts.conf /etc/icinga2/zones.d/master/hosts.conf
cp $icinga_backup/service.conf /etc/icinga2/zones.d/master/service.conf
cp $icinga_backup/influxdb.conf /etc/icinga2/features-available/influxdb.conf

influx -username <influxdb username> -password <influxdb password>
drop database icinga
create database icinga
use icinga
create retention policy "three_hour_only" on "icinga" duration 3h replication 1 default
exit

nodeshell all systemctl restart icinga2
systemctl restart icinga2

start LiCO 7.2 container

lico-control run  --image lico:7.2.0
lico-control docker-cmd remove lico_7.2.1

Upgrade from LiCO 7.2.0 to 7.2.1 for Container Deployment

Introduction

Prerequisites

Operating environment

Prepare

1. Stop LiCO Container

2.Prepare to upgrade

Step 1. Log in to the management node.

Step 2. Edit /root/lico_release.local and update the environment variables listed in that file:

Step 3. Save the changes to lico_release.local and reload the environment variables:

3. Build LiCO image

Step 1. Prepare packages. Download the following packages and upload all of them to the /root directory:

Step 2. Prepare build environment:

Step 3. Build LiCO image:

4. Backup

Upgrade software on management node

Step 1. Upgrade LiCO icinga2 plugin:

Step 2. Set and create a backup directory:

Step 3. Configure icinga global template file on management node:

Step 4. Define icinga zone file:

Step 5. Enable influxdb writer feature for icinga:

Step 6. Restart the icinga2 service to make the changes take effect.

Deploy LiCO 7.2 container

Step 1. Prepare the mount folder and configure the file

Step 2. merge the config file

Step 3. Edit the following configuration in the /etc/lico/lico.ini file

Step 4. Initial data for LiCO

Step 5. Start the Container

Rollback

Stop LiCO 7.2.1 container

restore database, config files and share dir

restore software in management node

start LiCO 7.2 container