You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 20 Next »

Job submission can be direct using the HTCondor batch system from the UIs or via Grid middleware from an UI that can be located anywhere.

In the latter case authentication and authorization are by default managed through Virtual Organizations (VO) [12]. To became member of the VO, the user needs a personal certificate and have to enroll on a VOMS Server.

Some examples of VOs hosted in CNAF VOMS servers (they are all replicated in another site, normally Padova or Naples) are:

Information about VOs can be found at:

Infrastructure

CNAF Tier1 has replaced LSF by HTCondor in the first half of 2020. The infrastructure consists of:

On a UI the user can submit a job to the submit node, which then deals with the routing to the central manager, responsible for dispatching the jobs to the worker nodes.

CEs in production:

HTCondor docs and guides are available here [6], [7].

HTCondor jobs

HTCondor is a job scheduler. You give HTCondor a file containing commands that tell it how to run jobs. HTCondor locates a machine that can run each job within the pool of machines, packages up the job and ships it off to this execute machine. The jobs run, and output is returned to the machine that submitted the jobs.

For HTCondor to run a job, it must be given details such as the names and location of the executable and all needed input files. These details are specified in the submit description file.

A simple example of an executable is a sleep job that waits for 40 seconds and then exits:

#!/bin/bash
# file name: sleep.sh

TIMETOWAIT="40"
echo "sleeping for $TIMETOWAIT seconds"
/bin/sleep $TIMETOWAIT

To submit this sample job, we need to specify all the details such as the names and location of the executable and all needed input files creating a submit description file where each line has the form

command name = value

like this:

# Unix submit description file
# sleep.sub -- simple sleep job

executable              = sleep.sh
log                     = sleep.log
output                  = outfile.txt
error                   = errors.txt
should_transfer_files   = Yes
when_to_transfer_output = ON_EXIT
queue

The main used commands to submit and manage jobs are:

  • condor_submit : Command for job submission. The following options are required:
    • -name sn-01.cr.cnaf.infn.it : to correctly address the job to the submit node
    • -spool : to transfer the input files and keep a local copy of the output files
    • the submission file (a .sub file containing the relevant information for the batch system), to be indicated as argument.
  • condor_q  :  Command to progressively check the job status.
    To be used always with the -name sn-01.cr.cnaf.infn.it option.
  • condor_transfer_data : Command to locally transfer the job output files once the execution is finished.
    To be used with -name sn-01.cr.cnaf.infn.it option followed by the cluster id returned by condor_submit command at submission.
  • condor_rm : Command to remove a job.
    To be used with -name sn-01.cr.cnaf.infn.it option followed by the cluster id returned by condor_submit command at submission.
  • condor_history : Command to see detailed information about all your jobs.

Submit local jobs

To submit jobs locally, i.e. from CNAF UI, use the following command:

condor_submit -spool -name sn-01.cr.cnaf.infn.it sleep.sub
-bash-4.2$ condor_submit -spool -name sn-01.cr.cnaf.infn.it sleep.sub
Submitting job(s).
1 job(s) submitted to cluster 6798880.

where 6798880 is the cluster id.

Submit Grid jobs

First, create the proxy:

voms-proxy-init --voms <vo name>

then submit the job with the following command:

export _condor_SEC_CLIENT_AUTHENTICATION_METHODS=GSI
condor_submit -pool ce02-htc.cr.cnaf.infn.it:9619 -remote ce02-htc.cr.cnaf.infn.it -spool sleep.sub

where "sleep.sub" is the submit file:

# Unix submit description file
# sleep.sub -- simple sleep job

+owner = undefined
executable = sleep.sh
log = sleep.log
output = outfile.txt
error = errors.txt
should_transfer_files = Yes
when_to_transfer_output = ON_EXIT
queue

that differs from the previous by the command name

+owner = undefined

that allows the computing element to identify the user through the voms-proxy.

Note that the script of the submit description file of a grid job is basically different of one that has to be submitted locally.

Monitoring

  • To see all jobs launched by a user:

    condor_q -submitter <user>

    Example:
    -bash-4.2$ condor_q -submitter ecorni
    -- Submitter: ecorni@htc_tier1 : <131.154.192.58:9618?... : sn-01.cr.cnaf.infn.it @ 10/25/19 09:55:57
    OWNER BATCH_NAME SUBMITTED DONE RUN IDLE HOLD TOTAL JOB_IDS
    ecorni ID: 6963 10/24 16:56 _ _ _ 1 1 6963.0
    ecorni ID: 6968 10/24 17:23 _ _ _ 1 1 6968.0
    ecorni ID: 8551 10/24 17:41 _ _ _ 1 1 8551.0
    ecorni ID: 8552 10/24 17:44 _ _ _ 1 1 8552.0
    ecorni ID: 8570 10/24 17:50 _ _ _ 1 1 8570.0

    Total for query: 5 jobs; 0 completed, 0 removed, 0 idle, 0 running, 5 held, 0 suspended
    Total for all users: 3937 jobs; 3880 completed, 0 removed, 9 idle, 0 running, 48 held, 0 suspended
    -bash-4.2$

  • To get the list of held jobs and the held reason:
    condor_q -submitter <user> -held
    Example:
    -bash-4.2$ condor_q -submitter ecorni -held
    -- Submitter: ecorni@htc_tier1 : <131.154.192.58:9618?... : sn-01.cr.cnaf.infn.it @ 10/25/19 09:56:46
    ID OWNER HELD_SINCE HOLD_REASON
    6963.0 ecorni 10/24 16:56 Failed to initialize user log to /home/TIER1/ecorni/logFile.log
    6968.0 ecorni 10/24 17:23 Failed to initialize user log to /home/TIER1/ecorni/logFile.log
    8551.0 ecorni 10/24 17:41 Failed to initialize user log to /home/TIER1/ecorni/logFile.log
    8552.0 ecorni 10/24 17:45 Failed to initialize user log to /home/TIER1/ecorni/logFile.log
    8570.0 ecorni 10/24 17:51 Failed to initialize user log to /home/TIER1/ecorni/logFile.log

    Total for query: 5 jobs; 0 completed, 0 removed, 0 idle, 0 running, 5 held, 0 suspended
    Total for all users: 3937 jobs; 3880 completed, 0 removed, 9 idle, 0 running, 48 held, 0 suspended
    -bash-4.2$

  • To get detailed information about a single job:
    condor_q -better-analyze -name sn-01.cr.cnaf.infn.it <cluster id>
    Example:
    -bash-4.2$ condor_q -better-analyze -name sn-01.cr.cnaf.infn.it 8570.0
    -- Schedd: sn-01.cr.cnaf.infn.it : <131.154.192.58:9618?...
    The Requirements expression for job 8570.000 is

    (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) &&
    (TARGET.Memory >= RequestMemory) && (TARGET.HasFileTransfer)

    Job 8570.000 defines the following attributes:

    DiskUsage = 1
    ImageSize = 1
    RequestDisk = DiskUsage
    RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)

    The Requirements expression for job 8570.000 reduces to these conditions:

    Slots
    Step Matched Condition
    ----- -------- ---------
    [0] 437 TARGET.Arch == "X86_64"
    [1] 437 TARGET.OpSys == "LINUX"
    [3] 437 TARGET.Disk >= RequestDisk
    [5] 437 TARGET.Memory >= RequestMemory
    [7] 437 TARGET.HasFileTransfer

    8570.000: Job is held.
    Hold reason: Failed to initialize user log to /home/TIER1/ecorni/logFile.log
    Last successful match: Thu Oct 24 17:51:15 2019

    8570.000: Run analysis summary ignoring user priority. Of 72 machines,
    0 are rejected by your job's requirements
    3 reject your job because of their own requirements
    0 match and are already running your jobs
    0 match but are serving other users
    69 are able to run your job
    -bash-4.2$

Retrieve the output files

The job outputs cannot be copied automatically. The user should launch:

condor_transfer_data -name sn-01.cr.cnaf.infn.it <cluster id>

NOTE: there is a limit of few MB on the size of files that can be transferred in this way. For larger file the data management tools have to be used. See the next chapter.

Example:

-bash-4.2$ condor_transfer_data -name sn-01.cr.cnaf.infn.it 8938
Fetching data files...
-bash-4.2$ ls -lhtr
total 0
-rw-r--r-- 1 ecorni tier1 173 Oct 25 14:35 test.sub
-rw-r--r-- 1 ecorni tier1 0 Oct 25 14:37 errorFile.err
-rw-r--r-- 1 ecorni tier1 1.1K Oct 25 14:37 logFile.log
-rw-r--r-- 1 ecorni tier1 34 Oct 25 14:37 outputFile.out

Another way is to have the output files written in a path shared between the WN and the UI. To do that, the user needs to modify the job executable as in the following example:

[fornaridampe@ui-tier1 htc_test]$ ls -lrth /storage/gpfs_data/dampe/users/fornaridampe/
total 0
[fornaridampe@ui-tier1 htc_test]$ cat htc_test.sub
universe = vanilla
executable = htc_test.sh
ShouldTransferFiles = YES
WhenToTransferOutput = ON_EXIT
queue 1
[fornaridampe@ui-tier1 htc_test]$ cat htc_test.sh
#!/bin/bash
hostname > /storage/gpfs_data/dampe/users/fornaridampe/output.txt 2>&1
[fornaridampe@ui-tier1 htc_test]$ condor_submit -name sn-01.cr.cnaf.infn.it -spool htc_test.sub
Submitting job(s).
1 job(s) submitted to cluster 461555.
[fornaridampe@ui-tier1 htc_test]$ condor_q -name sn-01.cr.cnaf.infn.it 461555
-- Schedd: sn-01.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 01/09/20 18:40:45
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
fornaridampe ID: 461555 1/9 18:40 _ _ 1 1 461555.0
Total for query: 1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended
Total for all users: 6629 jobs; 4477 completed, 7 removed, 1212 idle, 866 running, 67 held, 0 suspended
[fornaridampe@ui-tier1 htc_test]$ condor_q -name sn-01.cr.cnaf.infn.it 461555
-- Schedd: sn-01.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 01/09/20 18:41:53
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
fornaridampe ID: 461555 1/9 18:40 _ _ _ 1 461555.0
Total for query: 1 jobs; 1 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for all users: 6633 jobs; 4483 completed, 7 removed, 1211 idle, 865 running, 67 held, 0 suspended
[fornaridampe@ui-tier1 htc_test]$ ls -lrth /storage/gpfs_data/dampe/users/fornaridampe/
total 0
-rw-r--r-- 1 fornaridampe dampe 34 Jan 9 18:41 output.txt
[fornaridampe@ui-tier1 htc_test]$ cat /storage/gpfs_data/dampe/users/fornaridampe/output.txt
wn-201-07-33-01-a.cr.cnaf.infn.it
[fornaridampe@ui-tier1 htc_test]$

Submit grid jobs

In order to submit Grid jobs, the use use the middleware for computing grid (gLite)[12]. You can choose whether submit a job directly to the computing element (CE) or via the workload management system (WMS), however the gLite WMS support has been discontinued so that service is no longer available, at least in the Italian infrastructure.

In order to do it, the user needs to prepare a job file using the job described language (JDL) syntax[13]. A valid X.509 proxy certificates, hereafter proxy, must also be created.

To avoid failure on proxy expiration, it is very important to foreseen a proxy renewal mechanism. The proxy renewal mechanism is based on proxy delegation. First a long-term proxy should be created and stored on a dedicated server (myproxy.cnaf.infn.it for CNAF). This long-term proxy is used to create new short-term proxy before they expire.

Job submission directly through CE (glite-ce-job-submit)

First, we should create the proxy with the proper voms extension
voms-proxy-init --voms <vo name>
Then, we delegate our proxy credentials
glite-ce-delegate-proxy -e ce05-lcg.cr.cnaf.infn.it test

  • -e specifies the hostname and port number of the service (CE) where the user credentials have to be delegated.
  • test is the delegationID. The delegationID is the string identifying the proxy delegation and it should be used afterward.

Now, we have to submit job using the delegationID with the option -D
glite-ce-job-submit -D test -r ce05-lcg.cr.cnaf.infn.it:8443/cream-lsf-xenon glite-submit.jdl

  • -D test forces the submitted job to use previously delegated (via the command glite-ce-delegate-proxy) user credentials, identified with the specified ID.
  • specifies that the job has to be submitted to the CREAM CE identified by the specified ID.
    Standard format for the ID is:
    <full hostname>:<port number>/cream<service>-<queue name>
    where <service> specifies the underlying local resource management system (e.g. lsf, pbs, etc.)
  • glite-submit.jdl is the jdl file for the job




In the standard output the command return the ID of the job which is something like this:
*https://ce05-lcg.cr.cnaf.infn.it:8443/CREAM829568392*
In order to renew delegations and therefore renews the proxy of the jobs previously submitted to CREAM based CEs using these delegations, we should use the following command [14]:
glite-ce-proxy-renew -e ce05-lcg.cr.cnaf.infn.it test



  • -e ce05-lcg.cr.cnaf.infn.it selects the endpoint of the CREAM CE to send the proxy renewal request to.
    The endpoint must have the format <host>:<port> or <host>. This option is mandatory.
  • test is the delegationID.



This is a minimal JDL example:
bash$ cat ce.jdl
[
JobType = "Normal";
Executable = "/usr/bin/whoami";
StdOutput="out.out";
StdError="err.err";
OutputSandboxBaseDestUri="gsiftp://darkstorm.cnaf.infn.it/tmp/";
]



To request more job slots, even if the executable is single core, it is important to specify the granularity to have all the slots on the node, specifying these options in the JDL file:

CpuNumber = 4;
SMPGranularity = 4;

In this case the job will use 4 cores on the same node.

Other useful command are (see the glite user guide or the linux man for details):

  • glite-ce-job-cancel
  • glite-ce-job-status
  • glite-ce-job-output

Singularity in batch jobs

All the CNAF computing servers support containerization through Singularity[24] . Singularity is a containerization tool for running software in a reproducible way on various different platforms. With the advent of new operating systems, programming languages, and libraries, containers offer a sustainable solution for running old software. The software is shipped in so-called image, i.e. a file or folder containing a minimal operating system, the application to run and all its dependencies.
This section of the Tier-1 User Guide is intended to drive the user towards a transition from a native application workflow to one based on containers.
Singularity supports several image formats:

  • Singularity .img files
  • Singularity images in registry via the shub:// protocol
  • Docker images in registry via the docker:// protocol
  • a tar archive, eventually bzipped or gzipped
  • a folder

Obtain images

Official images for official software may have already been prepared by the software group within your experiment and be available through a shared filesystem (such as CVMFS), SingularityHub or other supported repository. Please check with your software manager about the support to Singularity.

Create a new image using a recipe (expert users)

A singularity recipe is a text file that contains all the instructions and configurations needed to build an image. It is organized in a header and a number of optional sections. The header is composed by a list of configuration settings in the form of "keyword: value" describing the system to use as base for the new image. Sections are identified by the % sign followed by a keyword. For a detailed list of all possible sections refer to[27].
An example of recipe file follows:

## Assume the file is called Recipe.txt and the final image is named name.img

# Headers
BootStrap: docker
From: cern/slc6-base

## Help section
%help
This text will be available by running

singularity help name.img

%labels
## Metadata to store (available through "singularity inspect name.img")
AUTHOR Carmelo Pellegrino
VERSION 1.0

## Environment variables setup
%environment
export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH

## Files to be copied in image from host system
%files
# source (in host system) [destination (in container, default "/")]
./code/ /

## Commands to execute in container during image building, after bootstrap
%post
yum -y update && yum -y upgrade && \
yum -y install --setopt=tsflags=nodocs \
devtoolset-6 compat-gcc-34-g77 \
cernlib-g77-devel cernlib-g77-utils

## Commands to be executed in default container instantiation
%runscript
exec /bin/bash


Build it with:

sudo singularity build name.img Recipe.txt

NOTE: since building a new image requires root access on the host system, it will not be possible on CNAF computing resources.

Run software

Singularity allows to run image-shipped software by simply prepending the singularity exec image.img text on the command line. The current working directory (CWD) is maintained and non-system files are available, so that, for example, the output of the ls command run in a container is not very different from the same command run natively.

Suppose the g77 compiler version 3.4 is contained in the name.img image. Example commands and their output follow:

$ cat hw.f
      program hello_world
      implicit none
      write ( *, * ) 'Hello, world!'
      stop
      end 
 $ singularity exec name.img g77 --version # i.e.: "g77 --version" runs in a container
GNU Fortran (GCC) 3.4.6 20060404 (Red Hat 3.4.6-19.el6)
Copyright (C) 2006 Free Software Foundation, Inc. 
GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING
or type the command `info -f g77 Copying'. 
 $ singularity exec name.img g77 hw.f -o hw
 $ singularity exec name.img ./hw
  Hello, world!

If a particular external folder is not available in container, for example exp_software or cvmfs, it can be forcedly bound by means of the -B orig:dest command line option. Many folders can also be bound at the same time:

$ singularity exec -B /cvmfs/:/cvmfs/ -B /opt/exp_software/:/opt/exp_software/ name.img ./hw

How to run Geant4 using Singularity

Geant4 needs old verion of “liblzma” and “libpcre” libraries related to Scien-tific Linux 6, for this reason an SL6 environment (using Singularity container) is therefore required to compile/execute the software from there.

Recompile Geant4 inside a container instantiated from a Singularity image with SL6 (usually provided by CNAF User Support), so that the libraries used are actually those that require Geant4. To do this, just start a Singularity container with the following command:

$ singularity exec /opt/exp_software/cupid/UI_SL6.img /bin/bash
Singularity>

then in the folder where the Geant sources are present, they can be recompiled.

If it is necessary to be able to access a host operating system path, simply use the ``-B'' (binding path) option specifying the host path separated from the mountpoint in the container using the colons. For example, to access /opt/exp_software/cupid, change the previous command to:

$ singularity exec -B /opt/exp_software:/opt/exp_software /opt/exp_software/cupid/UI_SL6.img /bin/bash
Singularity>

Once Geant has been recompiled, to submit job with bsub, the executable must be written in order to launch the commands related Geant within the Singularity container.

To perform, for example, the “cuspide” application, a line like the following must be inserted in the script:

singularity exec --no-home -B /opt/exp_software:/opt/exp_software /opt/exp_software/cupid/UI_SL6.img /opt/exp_software/cupid/geant4/bin/10.04.p02/cuspide

where the ``--no-home'' option causes Singularity to avoid mounting the user's home, since the job, not having the permissions to do so, would fail if it did.


  • No labels