The problem
Have you ever tried to install/use a software on Mistral and seen a message like this?
/lib64/libc.so.6: version `GLIBC_2.14` not found
This is for example one of the reasons why PyTorch is not available in our python3 module. Those software packages require a newer version of glibc
. Unfortunately, most of Mistral nodes are based on CentOS 6 kernel. To check the version of glibc
:
ldd --version
You will get an output like this:
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
So, how can you in some cases overcome this issue?
The workaround
Singularity containers
For many of the issues with glibc
, we propose the use of singularity
containers. The advantage is that you can use different Operating Systems running a higher version of glibc
per default. Example:
- CentOS 7
- Ubuntu 16
- ...
Prerequesites
- image with pytorch installed (you can also create it and then push it to Mistral)
- os with a kernel supported on Mistral
The PyTorch case
Let's take the simplest way:
- you search for a container image with pytorch installed (Dockerhub, Singularityhub)
- check if it works on Mistral (Kernel)
- load singularity module on MIstral
- pull the image
- run/execute the container
- use your software
Step (1) can be more complicated depending on your workflow and which other softwares are required: gpu, cuda, cudnn, nvidia drivers.
Here a concrete example:
% module load singularity/3.6.1-gcc-9.1.0
% singularity pull --name pytorch_gpu.simg shub://yinglilu/pytorch_gpu_singularity:1.3.0
INFO: Use cached image
% singularity shell pytorch_gpu.simg
Singularity> python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda custom (64-bit) on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>
If you check the version of glibc
inside the container, you will notice it has changed to a higher version.
Singularity> ldd --version
ldd (Debian GLIBC 2.24-11+deb9u4) 2.24
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
Singularity> uname -r
2.6.32-754.29.2.el6.x86_64
Singularity> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
The Caffe framework case
Another kernel related issue occurred when trying to build the [Caffe deep learning framework] on Mistral. Hopefully, they have some Docker images which are supported on Mistral (after conversion to singularity).
Example with the cpu version:
% module load singularity/3.6.1-gcc-9.1.0
% singlularity pull docker://bvic/caffe:cpu
% singularity run caffe_cpu.sif
Final remarks
Assumption
In this article, I assume that you are already familiar with the concept of container images and virtualization.
Important
Please take a look here for more information on how to use singularity on Mistral.
Note
Currently, containers are seen as workaround for some issues on Mistral. So, do not expect they will solve all problems.