Import system images

System-level container images are used by all users. You can obtain images from Lenovo salesperson or import images into LiCO as system-level container images. This section describes how to create and import system-level container images.

Image bootstrap files

LiCO is released with image bootstrap files for commonly-used AI frameworks. The image bootstrap files for commonly-used AI frameworks in the compressed package you obtained(https://hpc.lenovo.com/lico/downloads/6.1/images/k8s/image_bootstrap.zip). Users can use these files to create images.

The table below lists the image bootstrap files.

File name	Images	Framework	CPU/GPU	Comments
caffe-1.0-cpu	caffe:caffe-1.0-cpu	Caffe	CPU
caffe-1.0-gpu-cuda92	caffe:caffe-1.0-gpu-cuda92	Caffe	CUDA 9.2	Supports P100 and V100 Caffe does not support CUDA 9.0 officially
chainer-6.7.0-gpu-cuda101	chainer:chainer-6.7.0-gpu-cuda101	Chainer	CUDA 10.1	Supports P100, V100, RTX5000, RTX8000 and T4
intel-caffe-1.1.3-cpu	intel-caffe:intel-caffe-1.1.3-cpu	Intel-caffe	CPU
intel-python	intel-python:intel-python	Other	CPU
jupyter-py36	jupyter:jupyter-py36	Jupyter	CUDA 10.0	Supports P100, V100, RTX5000, RTX8000 and T4
jupyter-py37	jupyter:jupyter-py37	Jupyter	CUDA 11.0	Supports P100, V100, RTX5000, RTX8000 , T4 and A100
letrain-1.3.1-cuda110	lico:letrain-1.3.1-cuda110	LeTrain	CPU
lico-ai-scripts	lico-k8s-client:latest	Other	CPU	Indispensable
lico-file-manager	lico-file-manager:latest	Other	CPU	Indispensable
mxnet-1.5.0-cpu-mkl	mxnet:mxnet-1.5.0-cpu-mkl	Mxnet	CPU
mxnet-1.5.0-gpu-mkl-cuda100	mxnet:mxnet-1.5.0-gpu-mkl-cuda100	Mxnet	CUDA 10.0	Supports P100, V100, RTX5000, RTX8000 and T4
neon-2.6-cpu	neon:neon-2.6-cpu	Neon	CPU
pytorch-1.7.0-cuda110	pytorch:pytorch-1.7.0-cuda110	PyTorch	CUDA 11.0	Supports P100, V100, RTX5000, RTX8000, T4 and A100
scikit-single-cpu	scikit:scikit-single-cpu	Scikit	CPU
tensorflow-1.15.3-cuda110	tensorflow:tensorflow-1.15.3-cuda110	TensorFlow	CPU CUDA 11.0	Supports Keras(2.2.4) Supports P100, V100, RTX5000, RTX8000, T4 and A100
tensorflow-1.15.3-cuda110-hbase	tensorflow:tensorflow-1.15.3-cuda110-hbase	TensorFlow	CPU CUDA 11.0	Supports HBase Supports P100, V100, RTX5000, RTX8000 and T4
tensorflow-1.15.3-cuda110-keras	tensorflow:tensorflow-1.15.3-cuda110-keras	TensorFlow	CPU CUDA 11.0	Supports Keras(2.2.4) Supports P100, V100, RTX5000, RTX8000, T4 and A100
tensorflow-1.15.3-cuda110-mongodb	tensorflow:tensorflow-1.15.3-cuda110-mongodb	TensorFlow	CPU CUDA 11.0	Supports MongoDB Supports P100, V100, RTX5000, RTX8000, T4 and A100
tensorflow-1.15.2-mkl	tensorflow:tensorflow-1.15.2-mkl	TensorFlow	CPU CUDA 10.0	Supports P100, V100, RTX5000, RTX8000 and T4
tensorflow-2.2.0-cuda110	tensorflow:tensorflow-2.2.0-cuda110	TensorFlow	CPU CUDA 11.0	Supports P100, V100, RTX5000, RTX8000, T4 and A100

Create images

Step 1. Check to ensure that the lico-auth-internal.key file is under /etc/lico on the LiCO node.

Step 2. Prepare a build node with a minimum storage of 100 GB.

Notes:

The build node should be installed with docker.
Both opt/images and /var/tmp cannot be an NFS mount.
To create GPU images, GPU and GPU driver are required
To create GPU images, install nvidia-docker first. For more information, refer to https://github.com/NVIDIA/nvidia-docker.
To use images related to the MKL function, ensure that these images are created on a machine with the same CPU architecture.

Step 3. To the build node, copy the /etc/lico/lico-auth-internal.key file from the LiCO node. For example, copy the lico-auth-internal.key file to the new directory /opt/images. If the new directory cannot be found, create it manually.

step 4. To the build node, upload the compressed image bootstrap file you obtained which named image_bootstrap.zip(https://hpc.lenovo.com/lico/downloads/6.1/images/k8s/image_bootstrap.zip). For example, upload the compressed package to the new directory /opt/images.

Step 5. To the build node, run the following commands to configure the image:


cd /opt/images
unzip image_bootstrap.zip
cp /opt/images/lico-auth-internal.key /opt/images/image_bootstrap/lico-file-manager
chmod 744 /opt/images/image_bootstrap/lico-file-manager/lico-auth-internal.key

Step 6. To the build node, do one of the following to create image.

Run the following commands to create all images at once:


xxxxxxxxxx
cd /opt/images/image_bootstrap
make all

Run the following commands to create a group images at once:


xxxxxxxxxx
cd /opt/images/image_bootstrap
make caffe
make intel-caffe
make intel-python
make tensorflow
make neon
make chainer
make mxnet
make pytorch
make letrain
make jupyter
make lico-file-manager
make lico-k8s-client
make scikit

Step 7. Push the created docker image to one existing docker repository, the repository can be one docker registry (https://docs.docker.com/registry/) or one docker harbor (https://goharbor.io/) or docker hub (https://hub.docker.com/), just make sure the k8s nodes can access the docker repository. For example:


xxxxxxxxxx
docker tag lico-k8s-client:latest 10.240.212.106:5000/lico-k8s-client:latest
docker push 10.240.212.106:5000/lico-k8s-client:latest
...

Import images into LiCO as system-level images

Run the following commands on the LiCO node to import images from docker repository:


x
lico import_image kube-tools 10.240.212.106:5000/lico-k8s-client other
lico import_image lico-file-manager 10.240.212.106:5000/lico-file-manager other
lico import_image caffe-cpu 10.240.212.106:5000/caffe:caffe-1.0-cpu caffe
lico import_image caffe-gpu 10.240.212.106:5000/caffe:caffe-1.0-gpu-cuda92 caffe
lico import_image pytorch 10.240.212.106:5000/pytorch:pytorch-1.7.0-cuda110 pytorch
lico import_image letrain 10.240.212.106:5000/lico:letrain-1.3.1-cuda110 letrain
lico import_image jupyter-py36 10.240.212.106:5000/jupyter:jupyter-py36 jupyter -t py36 -t cpu -t gpu
lico import_image jupyter-py37 10.240.212.106:5000/jupyter:jupyter-py37 jupyter -t py37 -t cpu -t gpu
lico import_image scikit-cpu 10.240.212.106:5000/scikit:scikit-single-cpu scikit
lico import_image tensorflow2 10.240.212.106:5000/tensorflow:tensorflow-2.2.0-cuda110 tensorflow2
lico import_image tensorflow 10.240.212.106:5000/tensorflow:tensorflow-1.15.3-cuda110 tensorflow
lico import_image tensorflow-mkl 10.240.212.106:5000/tensorflow:tensorflow-1.15.2-mkl tensorflow
lico import_image tensorflow-hbase 10.240.212.106:5000/tensorflow:tensorflow-1.15.3-cuda110-hbase tensorflow
lico import_image tensorflow-keras 10.240.212.106:5000/tensorflow:tensorflow-1.15.3-cuda110-keras tensorflow
lico import_image tensorflow-mongodb 10.240.212.106:5000/tensorflow:tensorflow-1.15.3-cuda110-mongodb tensorflow
lico import_image intel-caffe 10.240.212.106:5000/intel-caffe:intel-caffe-1.1.3-cpu intel-caffe
lico import_image intel-python 10.240.212.106:5000/intel-python:intel-python other
lico import_image chainer-gpu 10.240.212.106:5000/chainer:chainer-6.7.0-gpu-cuda101 chainer
lico import_image mxnet-cpu 10.240.212.106:5000/mxnet:mxnet-1.5.0-cpu-mkl mxnet
lico import_image mxnet-gpu 10.240.212.106:5000/mxnet:mxnet-1.5.0-gpu-mkl-cuda100 mxnet
lico import_image neon-cpu 10.240.212.106:5000/neon:neon-2.6-cpu neon

Attention: Modify 10.240.212.106:5000 to the actual url of your docker repository.