Elementary Guide to Platform Agnostic Development

Date: 28 March 2019

Prepared by: Tomas Lazauskas

Premise

The aim of this notebook is to investigate containers as a flexible universal tool that would allow seamless transition from the laptop, to cloud, to HPC.

Proposed workflow

To achieve this, we aim to utilise existing tools with particular attention to those adaptable to various platforms, such as well-known containerization software Docker and Singularity.

Suppose we have a script vae.py which we intend to develop locally and then perform its production runs on Azure cloud and JADE and CSD3 UK’s national Tier 2 platforms utlising GPUs.

The vae.py script depends on the following packages:

• Python 2
• TensorFlow package with GPU support
• TensorFlow Probability Python library
• CUDA 9.0 / cuDNN 7.0

While developing locally, traditionally, we would install all the dependencies and use a version control tool, such as git (GitHub) to keep track of the process. However, the development process becomes challenging when the development environment is not exactly the same as the production environment, or there are multiple developers which are likely to use different OS, system libraries or language runtimes.

Docker

Docker has become an industry stardard to address the mentioned development/production challenges, especially for micro-service virtualization. Developers/users do not need to install project specific language environments and ideally what they would require is just Docker and the Dockerfile for the project.

The Dockerfile for the vae.py example script might look as follows:

# source of the base image
FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

# install environment dependencies
RUN apt-get update; apt-get -y install git python-setuptools python-dev build-essential
RUN easy_install pip

# install TensorFlow with GPU support
RUN pip install tensorflow-gpu

# install TensorFlow Probability library
RUN mkdir -p /usr/local/bin; cd /usr/local/bin; git clone -b r0.5 --single-branch https://github.com/tensorflow/probability.git
ENV PYTHONPATH="$PYTHONPATH:/usr/local/bin/probability"

# set enviromental parameters
ENV DYLD_LIBRARY_PATH="/usr/local/cuda/lib:$DYLD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"

# add 'TMPDIR' directory for vae.py
RUN mkdir -p /usr/local/tmp
ENV TMPDIR="/usr/local/tmp"

# add the vae.py script that is being developed
ADD vae.py /usr/local/bin/

# execute the script
CMD ["python", "/usr/local/bin/vae.py"]

The Dockerfile is based on the official base image provided by Nvidia which includes CUDA 9.0 and cuDNN 7.0 and we only need to install the rest of dependencies. We can use the following commands to:

• build an image: docker build -t tomaslaz/vae:latest .
• run the image: docker run -t tomaslaz/vae:latest
• pull the existing image: docker pull tomaslaz/vae:latest
• push the locally built image: docker push tomaslaz/vae:latest

More on the Docker’s base commands can be found here.

Using Docker for this particuar example allows us to almost seamlessly switch from the laptop(s) to other platforms, such as cloud, where all the users having access to the machine are trusted. However, Docker does not meet the security requirements for the general scietific computing usage. From an IT security perspective, a machine can be considered compromised if any user is able to run arbitrary code as the root use.

There is a fatal design flaw that limits Docker’s ability to run in HPC environments: for every container that Docker runs, the container process is spawned as a child of a root owned Docker daemon. As the user is able to directly interact with and control the Docker daemon, it is theoretically possible to coerce the daemon process into granting the users escalated privileges. Any user being able to escalate up to system administrator status, a user called “root”, would introduce unthinkable security risks for a shared compute environment.

Singularity

Amongst the solutions for HPC that address the Docker’s security issue is Singularity, which aims to make the support of existing and traditional HPC resources as easy as intsalling a single package on the host OS, and work in a very similar fashion as Docker.

Singularity works by compiling entire container into a single distributable file which in turn can be snapshotted, archived, and locked down such that it can be used later and the user can be confident that the code within the container has not changed. It seamlessly integrates with resource managers, such as SLURM, and natively supports technologies, such as InfiniBand and Lustre.

To build Singularity image from a Docker image is potentialy very easy - it requires downloading Docker image layers from DockerHub and assembling them into a Singularity container. For our example we could do it with the following command:

sudo singularity build vae.simg docker://tomaslaz/vae:latest

This image then can be copied on to HPC platforms that we would like to execute the production runs, for example using a submission script:

#!/bin/bash

# set the number of nodes
#SBATCH --nodes=1

# set max wallclock time
#SBATCH --time=1:00:00

# set name of job
#SBATCH --job-name=sing_test

# set number of GPUs
#SBATCH --gres=gpu:4

# mail alert at start, end and abortion of execution
#SBATCH --mail-type=ALL

# queue
#SBATCH --partition=big

module load singularity
module unload cuda/8.0
module load cuda/9.0

export NV_GPU=$CUDA_VISIBLE_DEVICES
export SINGULARITYENV_CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES

nvidia-smi
echo $CUDA_VISIBLE_DEVICES

# run the application
time singularity exec --nv -B /tmp:/tmp vae.simg python /usr/local/bin/vae.py -max_steps=2001 -batch_size=500

By using all the guidelines written here, we were able to perform bencharmking production runs on three different systems (Azure, CSD3, JADE) seamlessly - without aditional effort preparing specialized environment for the vae.py script.

Example use case - Benchmarking vae.py on multiple platforms

Notes and good practices for using containers:

Good practices:

• Use (only) official public images when possible (to control what is inside of your image),
• Optimise build cache (for example: add source code (or code that is most likely to change) as late as possible),
• Remove unnecessary tools (build the smallest image possible),
• Properly tag images,

Notes for Docker

• In order to use Docker with GPU support, Nvidia docker is required.

Notes for Singularity

• At the moment Singularity can run (only) on host Linux distributions, however, there is work done on Singularity Desktop for macOS (Alpha Preview), which allows Linux-based Singularity containers to be designed, built, tested, and signed/verified on macOS,
• MPI: MPI version on the host should match the MPI (major?) version in the container. (With Singularity, the MPI usage model is to call ‘mpirun’ from outside the container, and reference the container from your ‘mpirun’ command. )
• –nv is needed to enable native GPU support
• Singularity by default binds the following paths from the host system: $HOME /tmp /proc /sys /dev $PWD
• User files in the container by the default should be kept in: /usr/local/ /usr/bin/
• Make sure that the Linux versions in the container and on the Host’s are compatible.

References:

- https://cloud.google.com/solutions/best-practices-for-building-containers
- https://medium.com/travis-on-docker/why-and-how-to-use-docker-for-development-a156c1de3b24
- https://docs.docker.com/engine/reference/commandline/cli
- https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459
- https://www.nextplatform.com/2018/05/04/hpc-container-security-fact-myth-rumor-and-kernels/
- https://www.sylabs.io/singularity-desktop-macos/
- https://github.com/NVIDIA/nvidia-docker
- https://github.com/sylabs/singularity/issues/165