Singularity/Apptainer in batch jobs

All the Tier-1 computing servers support containerization through Apptainer [24] . Apptainer is a containerization tool for running software in a reproducible way on various different platforms. With the advent of new operating systems, programming languages, and libraries, containers offer a sustainable solution for running old software. The software is shipped in so-called image, i.e. a file or folder containing a minimal operating system, the application to run and all its dependencies.
This section of the Tier-1 User Guide is intended to drive the user towards a transition from a native application workflow to one based on containers.
Apptainer supports several image formats:

• Singularity .img files
• Singularity images in registry via the shub:// protocol
• Docker images in registry via the docker:// protocol
• a tar archive, eventually bzipped or gzipped
• a folder

Obtain images

Official images for official software may have already been prepared by the software group within your experiment and be available through a shared filesystem (such as CVMFS), SingularityHub or other supported repository. Please check with your software manager about the support to Apptainer.

Create a new image using a recipe (expert users)

A singularity recipe is a text file that contains all the instructions and configurations needed to build an image. It is organized in a header and a number of optional sections. The header is composed by a list of configuration settings in the form of "keyword: value" describing the system to use as base for the new image. Sections are identified by the % sign followed by a keyword. For a detailed list of all possible sections refer to [25].
An example of recipe file follows:

## Assume the file is called Recipe.txt and the final image is named name.img

# Headers
BootStrap: docker
From: cern/cc7-base

## Help section
%help
This text will be available by running

apptainer help name.img

%labels
## Metadata to store (available through "singularity inspect name.img")
AUTHOR Carmelo Pellegrino
VERSION 2.0

## Environment variables setup
%environment
export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH

## Files to be copied in image from host system
%files
# source (in host system) [destination (in container, default "/")]
./code/ /

## Commands to execute in container during image building, after bootstrap
%post
yum -y update && yum -y upgrade && \
yum -y install --setopt=tsflags=nodocs \
devtoolset-11 compat-libf2c-34 compat-gcc-44-gfortran gfortran

## Commands to be executed in default container instantiation
%runscript
exec /bin/bash

Build it with:

apptainer build name.img Recipe.txt

Run software

Apptainer allows to run image-shipped software by simply prepending the apptainer exec image.img text on the command line. The current working directory (CWD) is maintained and non-system files are available, so that, for example, the output of the ls command run in a container is not very different from the same command run natively.

Suppose the gfortran compiler version 4.4 is contained in the name.img image. Example commands and their output follow:

$ cat hw.f
      program hello_world
      implicit none
      write ( *, * ) 'Hello, world!'
      stop
      end 
 $ apptainer exec name.img gfortran44 --version # i.e.: "gfortran44 --version" runs in a container
GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-8)
Copyright (C) 2010 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING

 $ apptainer exec name.img gfortran44 hw.f -o hw
 $ apptainer exec name.img ./hw
  Hello, world!

If a particular external folder is not available in container, for example exp_software or cvmfs, it can be forcedly bound by means of the -B orig:dest command line option. Many folders can also be bound at the same time:

$ apptainer exec -B /cvmfs/:/cvmfs/ -B /opt/exp_software/:/opt/exp_software/ name.img ./hw 

How to run Geant4 using Apptainer

Geant4 needs old verion of “liblzma” and “libpcre” libraries related to Scientific Linux 6, for this reason an SL6 environment (using Singularity container) is therefore required to compile/execute the software from there.

Recompile Geant4 inside a container instantiated from a Apptainer image with SL6 (usually provided by CNAF User Support), so that the libraries used are actually those that require Geant4. To do this, just start a Apptainer container with the following command:

$ apptainer exec /opt/exp_software/cupid/UI_SL6.img /bin/bash
Apptainer>

then in the folder where the Geant sources are present, they can be recompiled.

If it is necessary to be able to access a host operating system path, simply use the ``-B'' (binding path) option specifying the host path separated from the mountpoint in the container using the colons. For example, to access /opt/exp_software/cupid, change the previous command to:

$ apptainer exec -B /opt/exp_software:/opt/exp_software /opt/exp_software/cupid/UI_SL6.img /bin/bash
Apptainer>

Once Geant has been recompiled, to submit job, the executable must be written in order to launch the commands related Geant within the Apptainer container.

To perform, for example, the “cuspide” application, a line like the following must be inserted in the script:

apptainer exec --no-home -B /opt/exp_software:/opt/exp_software /opt/exp_software/cupid/UI_SL6.img /opt/exp_software/cupid/geant4/bin/10.04.p02/cuspide

where the ``--no-home'' option causes Apptainer to avoid mounting the user's home, since the job, not having the permissions to do so, would fail if it did.

Run job inside apptainer image

On our HTCondor cluster it is possible to specify a Apptainer image to run a job inside.
To specify which image to use the user can add the attribute SingularityImage in the submit file:

$ cat submit-ubuntu.sub
# Unix submit description file
# sleep.sub -- simple sleep job
+SingularityImage = "docker://ubuntu:latest"

Requirements = HasSingularity

batch_name              = Singularity-Test-Ubuntu
executable              = os-check.sh
arguments               = 60
log                     = $(batch_name).log.$(Cluster).$(Process)
output                  = $(batch_name).out.$(Cluster).$(Process)
error                   = $(batch_name).err.$(Cluster).$(Process)
should_transfer_files   = Yes
when_to_transfer_output = ON_EXIT


queue

it is also useful to set Requirements = HasSingularity to request that te job will execute on a worker-node that can run singularity. 

Specify image

As shown in the example above, the image can be retreived from DockerHub (i.e. docker://) but also from local filesystem (e.g. /cvmfs or /opt/exp_software) providing the absolute path to the image.
The container image can also be sent to the worker node as an input file, but be aware that if the size of input files exceeds 10MB the job will fail.

Bind paths

The cluster is configured to automatically mount the following paths at the start fo the apptainer container: /cvmfs, /storage, /opt/exp_software.

A user can specify some custom paths to mount using the attribute SingularityBind, it will accept a space-separated list of paths with the follownig syntaxes:

• /original/path:/container/path → this will mount the first path inside the container on the second path
• /original/path → this will mount the original path in the same directory inside the container

+SingularityBind = "/opt/exp_software/test:/test /cvmfs/test"