Import system images

System-level container images are used by all users. You can obtain images from Lenovo salesperson and import images into LiCO as system-level container images. This section describes how to create and import system-level container images.

Image bootstrap files

Download image_bootstrap.zip from https://hpc.lenovo.com/lico/downloads/7.1/images/image_bootstrap.zip, image_bootstrap.zip includes the following singularity bootstrap(definition) files, then we can use it to create images.

Create images

Step 1. Prepare a build node with a minimum storage of 100 GB.

Notes:

• This node should have the same OS version and singularity version with the nodes in the cluster.
• If you want to create GPU images, this node should have the same GPU and GPU driver as nodes in the cluster.
• To use images related to the MKL function, ensure that these images are created on a machine with the same CPU architecture.

Step 2. To the build node, ensure that squashfs-tools, libarchive, and make are installed.

Step 3. To the build node, upload the compressed image bootstrap file you obtained which named image_bootstrap.zip. For example, upload the compressed package to the new directory /opt/images. If the new directory cannot be found, create it manually. Note that both this new directory and /var/tmp cannot be an NFS mount.

Step 4. To the build node, run the following commands to extract the compressed package.

cd /opt/images
unzip image_bootstrap.zip

Step 5. To the build node, do one of the following to create image. The created image file is in the dist folder of the current directory.

• Run the following commands to create all images at once:

  cd /opt/images/image_bootstrap
  make all

• Run the following commands to create a group images at once:

  cd /opt/images/image_bootstrap
  make caffe
  make cvat
  make intel-caffe
  make intel-python
  make tensorflow
  make mxnet
  make neon
  make chainer
  make letrain
  make jupyter
  make pytorch
  make rstudio
  make scikit
  make tensorrt
  make intel-pytorch
  make intel-tensorflow
  make paddlepaddle

  Notes: Check the network when one of the following errors is displayed:

  • ERROR MANIFEST_UNKNOWN: manifest unknown
  • ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
  • urllib2.URLError: <urlopen error [Error 104] Connection reset by peer>

Import images into LiCO as system-level images

Step 1. Copy the created images to the management node.

Attention:

• The created image needs to be uploaded to your shared home directory

• The shared home directory is the value corresponding to the SHARE_DIR configuration item in /etc/lico/lico.ini.

• This section uses the value /home as an example

• /var/tmp cannot be an NFS mount

Step 2. Run the following commands to import images into LiCO:

$SHARE_DIR='/home'

docker exec -it lico lico import_image caffe-cpu $SHARE_DIR/caffe-1.0-cpu.image caffe

docker exec -it lico lico import_image caffe-gpu $SHARE_DIR/caffe-1.0-gpu-cuda102.image caffe

docker exec -it lico lico import_image NVCaffe $SHARE_DIR/NVCaffe-0.17.3-gpu-cuda102.image caffe

docker exec -it lico lico import_image intel-caffe $SHARE_DIR/intel-caffe-1.1.6-cpu.image intel-caffe

docker exec -it lico lico import_image intel-python $SHARE_DIR/intel-python-3.7.image other

docker exec -it lico lico import_image tensorflow $SHARE_DIR/tensorflow-1.15.5-cuda118.image tensorflow

docker exec -it lico lico import_image tensorflow-mkl $SHARE_DIR/tensorflow-1.15.2-mkl.image tensorflow

docker exec -it lico lico import_image tensorflow-hbase $SHARE_DIR/tensorflow-1.15.5-cuda118-hbase.image tensorflow

docker exec -it lico lico import_image tensorflow-keras $SHARE_DIR/tensorflow-1.15.5-cuda118-keras.image tensorflow

docker exec -it lico lico import_image tensorflow-mongodb $SHARE_DIR/tensorflow-1.15.5-cuda118-mongodb.image tensorflow

docker exec -it lico lico import_image tensorflow2 $SHARE_DIR/tensorflow-2.10.1-cuda118.image tensorflow2

docker exec -it lico lico import_image intel-tensorflow $SHARE_DIR/intel-tensorflow-1.15.2-cpu.image tensorflow

docker exec -it lico lico import_image intel-tensorflow2-cpu $SHARE_DIR/intel-tensorflow-2.10.0-cpu.image tensorflow2

docker exec -it lico lico import_image mxnet-cpu $SHARE_DIR/mxnet-1.9.1-cpu.image mxnet

docker exec -it lico lico import_image mxnet-gpu $SHARE_DIR/mxnet-1.9.1-gpu-cuda112.image mxnet

docker exec -it lico lico import_image neon $SHARE_DIR/neon-2.6-cpu.image neon

docker exec -it lico lico import_image chainer-gpu $SHARE_DIR/chainer-6.7.0-gpu-cuda101.image chainer

docker exec -it lico lico import_image letrain $SHARE_DIR/letrain-2.0.0-cuda118.image letrain

docker exec -it lico lico import_image letrain-xpu $SHARE_DIR/letrain-2.0.0-xpu.image letrain

docker exec -it lico lico import_image jupyter-default $SHARE_DIR/jupyter-default.image jupyter -t py38 -t cpu -t gpu

docker exec -it lico lico import_image jupyter-py37 $SHARE_DIR/jupyter-py37.image jupyter -t py37 -t cpu -t gpu

docker exec -it lico lico import_image jupyter-py38 $SHARE_DIR/jupyter-py38.image jupyter -t py38 -t cpu -t gpu

docker exec -it lico lico import_image jupyter-intel-py37 $SHARE_DIR/jupyter-intel-py37.image jupyter -t intel_py37 -t cpu

docker exec -it lico lico import_image pytorch $SHARE_DIR/pytorch-1.14.0-cuda118.image pytorch

docker exec -it lico lico import_image intel-pytorch-cpu $SHARE_DIR/intel-pytorch-1.13.0-cpu.image pytorch

docker exec -it lico lico import_image rstudio $SHARE_DIR/rstudio.image rstudio -t cpu -t gpu

docker exec -it lico lico import_image scikit $SHARE_DIR/scikit-single-cpu.image scikit

docker exec -it lico lico import_image tensorrt8 $SHARE_DIR/tensorrt-8.5.1.image tensorrt -t tensorrt

docker exec -it lico lico import_image cvat $SHARE_DIR/cvat-2.3.0.image other

docker exec -it lico lico import_image paddlepaddle $SHARE_DIR/paddle-2.4.1-cuda117.image paddlepaddle

# If you use the intel openapi module, please import the following image
docker exec -it lico lico import_image jupyter-intel-optimized-pytorch \
$SHARE_DIR/jupyter-intel-optimized-pytorch.image jupyter -t pytorch

docker exec -it lico lico import_image jupyter-intel-optimized-tensorflow \
$SHARE_DIR/jupyter-intel-optimized-tensorflow.image jupyter -t tensorflow2


rm -f $SHARE_DIR/*.image