How to import LiCO system images (For Container)

System-level container images are used by all users. You can obtain images from Lenovo salesperson and import images into LiCO as system-level container images. This section describes how to create and import system-level container images.

Image bootstrap files

Download image_bootstrap.zip from https://hpc.lenovo.com/lico/downloads/8.0.0/images/image_bootstrap.zip, image_bootstrap.zip includes the following singularity bootstrap(definition) files, then we can use it to create images.

Create images

Step 1. Prepare a build node with a minimum storage of 100 GB.

Notes:

• This node should have the same OS version and singularity version with the nodes in the cluster.
• If you want to create GPU images, this node should have the same GPU and GPU driver as nodes in the cluster.
• To use images related to the MKL function, ensure that these images are created on a machine with the same CPU architecture.

Step 2. To the build node, ensure that squashfs-tools, libarchive, and make are installed.

Step 3. To the build node, upload the compressed image bootstrap file you obtained which named image_bootstrap.zip. For example, upload the compressed package to the new directory /opt/images. If the new directory cannot be found, create it manually. Note that both this new directory and /var/tmp cannot be an NFS mount.

Step 4. To the build node, run the following commands to extract the compressed package.

cd /opt/images
unzip image_bootstrap.zip

Step 5. To the build node, do one of the following to create image. The created image file is in the dist folder of the current directory.

• Run the following commands to create all images at once:

  cd /opt/images/image_bootstrap
  make all

• Run the following commands to create a group images at once:

  cd /opt/images/image_bootstrap
  make caffe
  make intel-caffe
  make intel-python
  make intel-pytorch
  make intel-tensorflow
  make jupyter
  make jupyterlab
  make llama2
  make mxnet
  make neon
  make onnx
  make paddlepaddle
  make pytorch
  make rstudio
  make scikit
  make tensorflow
  make tensorrt
  make cvat
  make letrain

  Notes: Check the network when one of the following errors is displayed:

  • ERROR MANIFEST_UNKNOWN: manifest unknown
  • ReadTimeoutError: HTTPSConnectionPool(host=‘files.pythonhosted.org’, port=443): Read timed out.
  • urllib2.URLError: <urlopen error [Error 104] Connection reset by peer>

Import images into LiCO as system-level images

Step 1. Copy the created images to the management node.

Attention:

• The created image needs to be uploaded to your shared home directory

• The shared home directory is the value corresponding to the SHARE_DIR configuration item in /etc/lico/lico.ini.

• This section uses the value /home as an example

• /var/tmp cannot be an NFS mount

Step 2. Run the following commands to import images into LiCO:

$SHARE_DIR='/home'

docker exec -it lico lico import_image caffe-cpu \
$SHARE_DIR/caffe-1.0-cpu.image caffe

docker exec -it lico lico import_image NVCaffe \
$SHARE_DIR/NVCaffe-0.17.3-gpu-cuda102.image caffe

docker exec -it lico lico import_image intel-caffe \
$SHARE_DIR/intel-caffe-1.1.6-cpu.image intel-caffe

docker exec -it lico lico import_image intel-python \
$SHARE_DIR/intel-python-3.10.image other

docker exec -it lico lico import_image tensorflow \
$SHARE_DIR/tensorflow-1.15.5-cuda121.image tensorflow

docker exec -it lico lico import_image tensorflow-mkl \
$SHARE_DIR/tensorflow-1.15.2-mkl.image tensorflow

docker exec -it lico lico import_image tensorflow-hbase \
$SHARE_DIR/tensorflow-1.15.5-cuda121-hbase.image tensorflow

docker exec -it lico lico import_image tensorflow-keras \
$SHARE_DIR/tensorflow-1.15.5-cuda121-keras.image tensorflow

docker exec -it lico lico import_image tensorflow-mongodb \
$SHARE_DIR/tensorflow-1.15.5-cuda121-mongodb.image tensorflow

docker exec -it lico lico import_image tensorflow2 \
$SHARE_DIR/tensorflow-2.16.1-cuda125.image tensorflow2

docker exec -it lico lico import_image intel-tensorflow \
$SHARE_DIR/intel-tensorflow-1.15.2-cpu.image tensorflow

docker exec -it lico lico import_image intel-tensorflow2-cpu \
$SHARE_DIR/intel-tensorflow-2.10.0-cpu.image tensorflow2

docker exec -it lico lico import_image mxnet-cpu \
$SHARE_DIR/mxnet-1.9.1-cpu.image mxnet

docker exec -it lico lico import_image mxnet-gpu \
$SHARE_DIR/mxnet-1.9.1-gpu-cuda124.image mxnet

docker exec -it lico lico import_image neon \
$SHARE_DIR/neon-2.6-cpu.image neon

docker exec -it lico lico import_image letrain \
$SHARE_DIR/letrain-2.1.0-cuda120.image letrain

docker exec -it lico lico import_image letrain-xpu \
$SHARE_DIR/letrain-2.0.0-xpu.image letrain

docker exec -it lico lico import_image jupyter-default \
$SHARE_DIR/jupyter-default.image jupyter -t py310 -t cpu -t gpu

docker exec -it lico lico import_image jupyter-py311 \
$SHARE_DIR/jupyter-py311.image jupyter -t py311 -t cpu -t gpu

docker exec -it lico lico import_image jupyter-py312 \
$SHARE_DIR/jupyter-py312.image jupyter -t py312 -t cpu -t gpu

docker exec -it lico lico import_image jupyterlab-default \
$SHARE_DIR/jupyterlab-default.image jupyterlab -t py310 -t cpu -t gpu

docker exec -it lico lico import_image jupyterlab-py311 \
$SHARE_DIR/jupyterlab-py311.image jupyterlab -t py311 -t cpu -t gpu

docker exec -it lico lico import_image jupyterlab-py312 \
$SHARE_DIR/jupyterlab-py312.image jupyterlab -t py312 -t cpu -t gpu

docker exec -it lico lico import_image pytorch \
$SHARE_DIR/pytorch-2.4.0-cuda125.image pytorch

docker exec -it lico lico import_image intel-pytorch-cpu \
$SHARE_DIR/intel-pytorch-2.0.1-cpu.image pytorch

docker exec -it lico lico import_image rstudio \
$SHARE_DIR/rstudio.image rstudio -t cpu -t gpu

docker exec -it lico lico import_image scikit \
$SHARE_DIR/scikit-single-cpu.image scikit

docker exec -it lico lico import_image tensorrt8 \
$SHARE_DIR/tensorrt-8.5.3-cuda120.image tensorrt -t tensorrt

docker exec -it lico lico import_image cvat \
$SHARE_DIR/cvat-2.3.0.image other

docker exec -it lico lico import_image paddlepaddle \
$SHARE_DIR/paddle-2.6.0-cuda124.image paddlepaddle

docker exec -it lico lico import_image onnx \
$SHARE_DIR/onnx-1.19.2-cuda124.image onnx

docker exec -it lico lico import_image llama2-finetuning \
$SHARE_DIR/llama2-finetuning-cu123.image other -t llama

docker exec -it lico lico import_image llama2-testing \
$SHARE_DIR/llama2-testing-cu123.image other -t llama

rm -f $SHARE_DIR/*.image