Upgrade from LiCO 7.1.0 to 7.2.0 for Container Deployment

Prepare

1. Stop LiCO Container

lico-control stop
docker rename lico lico_7.1

2.Prepare to upgrade

Step 1. Log in to the management node.

Step 2. Edit /root/lico_release.local and update the environment variables listed in that file:

# Management node hostname
sms_name="head"
# IP address of management node
sms_ip="192.168.0.2"
# Old version of repository directory for LiCO and the version of LiCO in
# the current system
original_lico_repo_dir="/install/custom/lico-7.1.0"
# Old version of repository directory for LiCO-dep and
# the version of LiCO-dep in the current system
original_lico_dep_repo_dir="/install/custom/lico-dep-7.1.0"
# New version of repository directory for LiCO, please change it according to LiCO version
new_lico_repo_dir="/install/custom/lico-7.2.0"
# New version of LiCO repository directory for LiCO-dep,
# please change it according to LiCO-dep version
new_lico_dep_repo_dir="/install/custom/lico-dep-7.2.0"
# Link name of LiCO repository directory for LiCO
link_lico_repo_dir="/install/custom/lico"
# Link name of LiCO repository directory for LiCO-dep
link_lico_dep_repo_dir="/install/custom/lico-dep"
# Existing shared directories for LiCO
lico_share_dir="/home"
# Shared directories of repo file in LiCO Installation Guide
share_installer_dir="/install/installer"
# link name of directory for lico moniotr
link_lico_monitor_dir="/install/custom/lico-monitor"
# Local directory for for lico monitor, please change it according to this version.
lico_monitor_dir="/install/custom/lico-monitor-1.1.0"

Step 3. Save the changes to lico_release.local and reload the environment variables:

chmod 600 lico_release.local
source lico_release.local

3. Build LiCO image

Step 1. Prepare packages. Download the following packages and upload all of them to the /root directory:

• Lenovo-OpenHPC-2.6.2.EL8.x86_64.tar:

https://hpc.lenovo.com/lico/downloads/7.2/Lenovo-OpenHPC-2.6.2.EL8.x86_64.tar • lico-dep-7.2.0.el8.x86_64.tgz:

https://hpc.lenovo.com/lico/downloads/7.2/lico-dep-7.2.0.el8.x86_64.tgz • authselect.tar.gz:

https://hpc.lenovo.com/lico/downloads/7.2/examples/conf/authselect/authselect.tar.gz • LiCO 7.2.0 release package lico-release-7.2.0.el8.x86_64.tar.gz:

https://commercial.lenovo.com/cn/en/signin • LiCO 7.2.0 docker package lico-docker-7.2.0.x86_64.tar.gz:

https://commercial.lenovo.com/cn/en/signin

Step 2. Prepare build environment:

cd /root
lico_container_workspace="/root/lico-workspace/7.2.0"
mkdir -p $lico_container_workspace
tar -xzvf lico-docker-7.2.0.x86_64.tar.gz -C $lico_container_workspace
mv Lenovo-OpenHPC-2.6.2.EL8.x86_64.tar $lico_container_workspace
mv lico-dep-7.2.0.el8.x86_64.tgz $lico_container_workspace
mv lico-release-7.2.0.el8.x86_64.tar.gz $lico_container_workspace
mv authselect.tar.gz $lico_container_workspace

cp $lico_container_workspace/lico-docker/Dockerfile \
$lico_container_workspace/lico-docker/scripts/build.sh $lico_container_workspace

\cp $lico_container_workspace/lico-docker/scripts/lico-control /usr/bin/

chmod +x $lico_container_workspace/build.sh
chmod +x /usr/bin/lico-control

Step 3. Build LiCO image:

cd $lico_container_workspace
./build.sh

4. Backup

backup_dir=/root/lico_7.1_backup/
mkdir -p $backup_dir
cd $lico_container_workspace/lico-docker/upgrade
python3 container-upgrade backup --backup-path $backup_dir --backup-db --backup-config --backup-sharedir

Upgrade software on management node

Step 1. Upgrade LiCO icinga2 plugin:

systemctl stop icinga2
nodeshell all systemctl stop icinga2
rm -rf /opt/lico/pub/monitor
mkdir -p $lico_monitor_dir
tar -xvzf openlico-monitor-1.1.0.x86_64.tgz -C $lico_monitor_dir
rm -rf $link_lico_monitor_dir
ln -s $lico_monitor_dir $link_lico_monitor_dir
python3 $link_lico_monitor_dir/install.py

Step 2. Set and create a backup directory:

icinga_backup="/install/icinga2/backup"
mkdir -p $icinga_backup

Step 3. Configure icinga global template file on management node:

Backup the existing configuration files:

mv /etc/icinga2/zones.d/global-templates/commands.conf $icinga_backup

Download commands.conf from the following command and replace it to /etc/icinga2/zones.d/global-templates.
```
curl -O https://hpc.lenovo.com/lico/downloads/7.2/examples/conf/icinga2/commands/slurm/commands.conf
```

Modify the privileges

chown -R icinga:icinga /etc/icinga2/zones.d/global-templates

Step 4. Define icinga zone file:

Backup the existing configuration files:

cp /etc/icinga2/zones.d/master/hosts.conf $icinga_backup
mv /etc/icinga2/zones.d/master/service.conf $icinga_backup

Locate the configuration information of the Icinga 2 management node in the /etc/icinga2/zones.d/master/hosts.conf file, and set the configuration entry as vars.is_master = 1. Following is an example whose management node name is head.
```
object Host "head" {
   check_command = "hostalive"
   address = "192.168.0.1"
   vars.is_master = 1
   vars.agent_endpoint = name
}
```
Download service.conf from https://hpc.lenovo.com/lico/downloads/7.2/examples/conf/icinga2/ and upload it to /etc/icinga2/zones.d/master

Modify the privileges:

chown -R icinga:icinga /etc/icinga2/zones.d/master

Step 5. Enable influxdbwriter feature for icinga:

Backup the existing configuration files:

cp /etc/icinga2/features-available/influxdb.conf $icinga_backup

Download influxdb.conf from https://hpc.lenovo.com/lico/downloads/7.2/examples/conf/icinga2/ and upload it to /etc/icinga2/features-available/

Update the configuration according to the configuration items in the original configuration file:

host = "<influxdb server address>"
port = 8086 
database = "icinga" 
username = "<username>" 
password = "<password>"

Step 6. Restart the icinga2 service to make the changes take effect.

nodeshell all systemctl restart icinga2
systemctl restart icinga2

Deploy LiCO 7.2 container

Step 1. Prepare the mount folder and configure the file

rm -rf /etc/lico
lico-control prepare

Step 2. merge the config file

cp  $backup_dir/head/configure/lico/nodes.csv /etc/lico/
cd /root/lico-workspace/7.2.0/lico-docker/upgrade/
python3 container-upgrade merge-config --backup-path $backup_dir

Step 3. Initial data for LiCO

lico-control init --mode  lico

lico-control init --mode  cloudtools

Step 4. Start the Container

lico-control run 
lico-control docker-cmd remove lico_7.1

Rollback

Stop LiCO 7.2 container

lico-control stop
lico-control docker-cmd rename lico lico_7.2

cd $lico_container_workspace/lico-docker/upgrade/
python3 container-rollback --backup-path $backup_dir --restore-db --restore-sharedir

restore software in management node

systemctl stop icinga2
nodeshell all systemctl stop icinga2
rm -rf /opt/lico/pub/monitor
source lico_env.local 
mkdir -p $lico_monitor_dir
tar -xvzf openlico-monitor-1.0.0.x86_64.tgz -C $lico_monitor_dir
rm -rf $link_lico_monitor_dir
rm -rf $new_lico_monitor_dir
ln -s $lico_monitor_dir $link_lico_monitor_dir
python3 $link_lico_monitor_dir/install.py
nodeshell all systemctl start icinga2

start LiCO 7.1 container

lico-control run  --image lico:7.1.0
lico-control docker-cmd remove lico_7.2

Upgrade from LiCO 7.1.0 to 7.2.0 for Container Deployment

Prepare

1. Stop LiCO Container

2.Prepare to upgrade

Step 1. Log in to the management node.

Step 2. Edit /root/lico_release.local and update the environment variables listed in that file:

Step 3. Save the changes to lico_release.local and reload the environment variables:

3. Build LiCO image

Step 1. Prepare packages. Download the following packages and upload all of them to the /root directory:

Step 2. Prepare build environment:

Step 3. Build LiCO image:

4. Backup

Upgrade software on management node

Step 1. Upgrade LiCO icinga2 plugin:

Step 2. Set and create a backup directory:

Step 3. Configure icinga global template file on management node:

Step 4. Define icinga zone file:

Step 5. Enable influxdbwriter feature for icinga:

Step 6. Restart the icinga2 service to make the changes take effect.

Deploy LiCO 7.2 container

Step 1. Prepare the mount folder and configure the file

Step 2. merge the config file

Step 3. Initial data for LiCO

Step 4. Start the Container

Rollback

Stop LiCO 7.2 container

restore database, config files and share dir

restore software in management node

start LiCO 7.1 container