How to import LiCO system images

System-level container images are used by all users. You can obtain images from Lenovo salesperson or import images into LiCO as system-level container images. This section describes how to create and import system-level container images.

Image bootstrap files

Download image_bootstrap.zip from https://hpc.lenovo.com/lico/downloads/7.2.1/images/image_bootstrap.zip, image_bootstrap.zip includes the following singularity bootstrap(definition) files, then we can use it to create images.

Create images

Step 1. Prepare a build node with a minimum storage of 100 GB.

Notes:

• This node should have the same OS version and singularity version with the nodes in the cluster.
• If you want to create GPU images, this node should have the same GPU and GPU driver as nodes in the cluster.
• To use images related to the MKL function, ensure that these images are created on a machine with the same CPU architecture.

Step 2. To the build node, ensure that squashfs-tools, libarchive, and make are installed.

Step 3. To the build node, upload the compressed image bootstrap file you obtained which named image_bootstrap.zip. For example, upload the compressed package to the new directory /opt/images. If the new directory cannot be found, create it manually. Note that both this new directory and /var/tmp cannot be an NFS mount.

Step 4. To the build node, run the following commands to extract the compressed package.

cd /opt/images
unzip image_bootstrap.zip

Step 5. To the build node, do one of the following to create image. The created image file is in the dist folder of the current directory.

• Run the following commands to create all images at once:

  cd /opt/images/image_bootstrap
  make all

• Run the following commands to create a group images at once:

  cd /opt/images/image_bootstrap
  make caffe
  make intel-caffe
  make intel-python
  make intel-pytorch
  make intel-tensorflow
  make jupyter
  make jupyterlab
  make llama2
  make mxnet
  make neon
  make onnx
  make paddlepaddle
  make pytorch
  make rstudio
  make scikit
  make tensorflow
  make tensorrt
  make cvat
  make letrain

  Notes: Check the network when one of the following errors is displayed:

  • ERROR MANIFEST_UNKNOWN: manifest unknown
  • ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
  • urllib2.URLError: <urlopen error [Error 104] Connection reset by peer>

Import images into LiCO as system-level images

Step 1. Copy the created images to the management node.

For example, copy the images to directory /opt/images. Note that both this directory and /var/tmp cannot be an NFS mount

Step 2. Run the following commands to import images into LiCO:

cd /opt/images

lico import_image caffe-cpu caffe-1.0-cpu.image caffe

lico import_image NVCaffe NVCaffe-0.17.3-gpu-cuda102.image caffe

lico import_image intel-caffe intel-caffe-1.1.6-cpu.image intel-caffe

lico import_image intel-python intel-python-3.10.image other

lico import_image tensorflow tensorflow-1.15.5-cuda121.image tensorflow

lico import_image tensorflow-mkl tensorflow-1.15.2-mkl.image tensorflow

lico import_image tensorflow-hbase tensorflow-1.15.5-cuda121-hbase.image tensorflow

lico import_image tensorflow-keras tensorflow-1.15.5-cuda121-keras.image tensorflow

lico import_image tensorflow-mongodb tensorflow-1.15.5-cuda121-mongodb.image tensorflow

lico import_image tensorflow2 tensorflow-2.12.0-cuda121.image tensorflow2

lico import_image intel-tensorflow intel-tensorflow-1.15.2-cpu.image tensorflow

lico import_image intel-tensorflow2-cpu intel-tensorflow-2.10.0-cpu.image tensorflow2

lico import_image mxnet-cpu mxnet-1.9.1-cpu.image mxnet

lico import_image mxnet-gpu mxnet-1.9.1-gpu-cuda112.image mxnet

lico import_image neon neon-2.6-cpu.image neon

lico import_image letrain letrain-2.1.0-cuda120.image letrain

lico import_image letrain-xpu letrain-2.0.0-xpu.image letrain

lico import_image jupyter-default jupyter-default.image jupyter -t py38 -t cpu -t gpu

lico import_image jupyter-py39 jupyter-py39.image jupyter -t py39 -t cpu -t gpu

lico import_image jupyter-py310 jupyter-py310.image jupyter -t py310 -t cpu -t gpu

lico import_image jupyterlab-default jupyterlab-default.image jupyterlab -t py38 -t cpu -t gpu

lico import_image jupyterlab-py39 jupyterlab-py39.image jupyterlab -t py39 -t cpu -t gpu

lico import_image jupyterlab-py310 jupyterlab-py310.image jupyterlab -t py310 -t cpu -t gpu

lico import_image pytorch pytorch-2.1.0-cuda121.image pytorch

lico import_image intel-pytorch-cpu intel-pytorch-2.0.1-cpu.image pytorch

lico import_image rstudio rstudio.image rstudio -t cpu -t gpu

lico import_image scikit scikit-single-cpu.image scikit

lico import_image tensorrt8 tensorrt-8.5.3-cuda120.image tensorrt -t tensorrt

lico import_image cvat cvat-2.3.0.image other

lico import_image paddlepaddle paddle-2.4.0-cuda120.image paddlepaddle

lico import_image onnx onnx-1.17.1-cuda122.image onnx

lico import_image llama2-finetuning llama2-finetuning-cu123.image other -t llama

lico import_image llama2-testing llama2-testing-cu123.image other -t llama