Import system images

System-level container images are used by all users. You can obtain images from Lenovo salesperson or import images into LiCO as system-level container images. This section describes how to create and import system-level container images.

Image bootstrap files

Download image_bootstrap.zip from https://hpc.lenovo.com/lico/downloads/7.0/images/host/image_bootstrap.zip, image_bootstrap.zip includes the following singularity bootstrap(definition) files, then we can use it to create images.

Create images

Step 1. Prepare a build node with a minimum storage of 100 GB.

Notes:

• This node should have the same OS version and singularity version with the nodes in the cluster.
• If you want to create GPU images, this node should have the same GPU and GPU driver as nodes in the cluster.
• To use images related to the MKL function, ensure that these images are created on a machine with the same CPU architecture.

Step 2. To the build node, ensure that squashfs-tools, libarchive, and make are installed.

Step 3. To the build node, upload the compressed image bootstrap file you obtained which named image_bootstrap.zip. For example, upload the compressed package to the new directory /opt/images. If the new directory cannot be found, create it manually. Note that both this new directory and /var/tmp cannot be an NFS mount.

Step 4. To the build node, run the following commands to extract the compressed package.

cd /opt/images
unzip image_bootstrap.zip

Step 5. To the build node, do one of the following to create image. The created image file is in the dist folder of the current directory.

• Run the following commands to create all images at once:

  xxxxxxxxxx
  cd /opt/images/image_bootstrap
  make all
  

• Run the following commands to create a group images at once:

  xxxxxxxxxx
  cd /opt/images/image_bootstrap
  make caffe
  make cvat
  make intel-caffe
  make intel-python
  make tensorflow
  make mxnet
  make neon
  make chainer
  make letrain
  make jupyter
  make pytorch
  make rstudio
  make scikit
  make tensorrt
  make intel-pytorch
  make intel-tensorflow
  make paddlepaddle
  

  Notes: Check the network when one of the following errors is displayed:

  • ERROR MANIFEST_UNKNOWN: manifest unknown
  • ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
  • urllib2.URLError: <urlopen error [Error 104] Connection reset by peer>

Import images into LiCO as system-level images

Step 1. Copy the created images to the management node.

For example, copy the images to directory /opt/images. Note that both this directory and /var/tmp cannot be an NFS mount

Step 2. Run the following commands to import images into LiCO:

​x
cd /opt/images

lico import_image caffe-cpu caffe-1.0-cpu.image caffe

lico import_image caffe-gpu caffe-1.0-gpu-cuda92.image caffe

lico import_image NVCaffe NVCaffe-0.17.3-gpu-cuda102.image caffe

lico import_image intel-caffe intel-caffe-1.1.6-cpu.image intel-caffe

lico import_image intel-python intel-python-3.7.image other

lico import_image tensorflow tensorflow-1.15.3-cuda110.image tensorflow

lico import_image tensorflow-mkl tensorflow-1.15.2-mkl.image tensorflow

lico import_image tensorflow-hbase tensorflow-1.15.3-cuda110-hbase.image tensorflow

lico import_image tensorflow-keras tensorflow-1.15.3-cuda110-keras.image tensorflow

lico import_image tensorflow-mongodb tensorflow-1.15.3-cuda110-mongodb.image tensorflow

lico import_image tensorflow2 tensorflow-2.5.0-cuda114.image tensorflow2

lico import_image intel-tensorflow intel-tensorflow-1.15.2-cpu.image tensorflow

lico import_image intel-tensorflow2 intel-tensorflow-2.3.0-cpu.image tensorflow2

lico import_image mxnet-cpu mxnet-1.9.1-cpu.image mxnet

lico import_image mxnet-gpu mxnet-1.9.1-gpu-cuda112.image mxnet

lico import_image neon neon-2.6-cpu.image neon

lico import_image chainer-gpu chainer-6.7.0-gpu-cuda101.image chainer

lico import_image letrain letrain-1.7.0-cuda110.image letrain

lico import_image jupyter-default jupyter-default.image jupyter -t py38 -t cpu -t gpu

lico import_image jupyter-py37 jupyter-py37.image jupyter -t py37 -t cpu -t gpu

lico import_image jupyter-py38 jupyter-py38.image jupyter -t py38 -t cpu -t gpu

lico import_image jupyter-intel-py37 jupyter-intel-py37.image jupyter -t intel_py37 -t cpu

lico import_image pytorch pytorch-1.13.0-cuda118.image pytorch

lico import_image intel-pytorch intel-pytorch-1.7.0-cpu.image pytorch

lico import_image rstudio rstudio.image rstudio -t cpu -t gpu

lico import_image scikit-cpu scikit-single-cpu.image scikit

lico import_image tensorrt tensorrt-7.1.3.4.image tensorrt -t tensorrt

lico import_image tensorrt tensorrt-8.0.1.6.image tensorrt -t tensorrt

lico import_image cvat cvat-1.7.0.image other

lico import_image paddlepaddle paddlepaddle_2.3.0.image paddlepaddle

# If you use the intel openapi module, please import the following image
lico import_image jupyter-intel-optimized-pytorch \
jupyter-intel-optimized-pytorch.image jupyter -t pytorch

lico import_image jupyter-intel-optimized-tensorflow \
jupyter-intel-optimized-tensorflow.image jupyter -t tensorflow2