All the options and the submit description commands of the condor_submit command are available in the Command Reference Manual [26]. Also for a short guide on the submit description file and its commands you can see the Appendix A.
Some helpful examples follow below.
CPUs, GPUs and RAM requests
Generally, for a job it could be useful to specify the number of CPUs or maybe it would be better to specify the amount of required RAM with the options:
request_cpus = <number of CPUs>request_memory = <RAM amount in MB>
in the command lines of the job submit file. For example, this can be the script of a submit description file with specific requests of CPUs and RAM:
-bash-4.2$ cat sleep.sub # Unix submit description file # sleep.sub -- simple sleep job request_cpus = 2 request_memory = 1000 executable = sleep.sh log = sleep.log output = outfile.txt error = errors.txt should_transfer_files = Yes when_to_transfer_output = ON_EXIT queue
On the other hand, if your job has to use GPUs for running, you have to insert the right requirement:
+WantGPU = true request_GPUs = 1 requirements = (TARGET.CUDACapability >= 1.2) && (TARGET.CUDADeviceName =?= "Tesla K40m") && $(requirements:True)
Jobs with ROOT-program as executable
First of all, you have to setup ROOT before using it. Your collaboration may have installed a ROOT distribution in /opt/exp_software (which is a location shared between the user-interface and the worker nodes).
In this case you should find one or more ROOT installation directories there:
[fornaricta@ui-tier1]$ ls /opt/exp_software/cta/local_software/root/ 5.34.26 5.34.36 5.34.38 root root-6.10.08 root-6.16.00 root_build_5.34.38
so you can choose your preferred version with a submit file like the following one:
[fornaricta@ui-tier1]$ cat test.sub universe = vanilla executable = test.sh arguments = 5.34.26 output = job.out error = job.err log = job.log WhenToTransferOutput = ON_EXIT ShouldTransferFiles = YES queue 1
where the executable file has this content:
[fornaricta@ui-tier1]$ cat test.sh #!/bin/bash source /storage/gpfs_data/ctalocal/fornaricta/root_config.sh $1 /opt/exp_software/cta/local_software/root/$1/bin/root -b -q
and the configuration script (located on a gpfs path, shared between the user-interface and the worker nodes) is:
[fornaricta@ui-tier1]$ cat /storage/gpfs_data/ctalocal/fornaricta/root_config.sh #!/bin/bash export LD_LIBRARY_PATH=/opt/exp_software/cta/local_software/root/$1/lib/root:$LD_LIBRARY_PATH
Submitting:
[fornaricta@ui-tier1]$ condor_submit -spool -name sn-02.cr.cnaf.infn.it test.sub
Submitting job(s).
1 job(s) submitted to cluster 5824045.
[fornaricta@ui-tier1]$ condor_q -name sn-02.cr.cnaf.infn.it 5824045.0
-- Schedd: sn-02.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 07/08/20 18:54:17
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
fornaricta ID: 5824045 7/8 18:54 _ _ 1 1 5824045.0
Total for query: 1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended
Total for all users: 45425 jobs; 25623 completed, 3 removed, 13181 idle, 5297 running, 1321 held, 0 suspended
[fornaricta@ui-tier1]$ condor_q -name sn-02.cr.cnaf.infn.it 5824045.0
-- Schedd: sn-01.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 07/08/20 18:55:03
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
fornaricta ID: 5824045 7/8 18:54 _ 1 _ 1 5824045.0
Total for query: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended
Total for all users: 45474 jobs; 25644 completed, 3 removed, 13222 idle, 5286 running, 1319 held, 0 suspended
[fornaricta@ui-tier1]$ condor_q -name sn-02.cr.cnaf.infn.it 5824045.0
-- Schedd: sn-01.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 07/08/20 18:55:04
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
fornaricta ID: 5824045 7/8 18:54 _ _ _ 1 5824045.0
Total for query: 1 jobs; 1 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for all users: 45476 jobs; 25646 completed, 3 removed, 13223 idle, 5284 running, 1320 held, 0 suspended
[fornaricta@ui-tier1]$ condor_transfer_data -name sn-02.cr.cnaf.infn.it 5824045.0
Fetching data files...
[fornaricta@ui-tier1]$ cat job.err
[fornaricta@ui-tier1]$ cat job.out
*******************************************
* *
* W E L C O M E to R O O T *
* *
* Version 5.34/26 20 February 2015 *
* *
* You are welcome to visit our Web site *
* http://root.cern.ch *
* *
*******************************************
ROOT 5.34/26 (v5-34-26@v5-34-26, Jun 16 2015, 18:41:55 on linuxx8664gcc)
CINT/ROOT C/C++ Interpreter version 5.18.00, July 2, 2010
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }
If no ROOT installation is available in /opt/exp_software, you can source one of the multiple distributions available from CVMFS:
[fornarivirgo@ui01-virgo root_test]$ ls /cvmfs/sft.cern.ch/lcg/releases/ROOT/ 5.34.24-64287 6.06.06-71859 6.10.00-8b404 6.12.04-4473c 6.12.06-76fef 6.14.00-66c89 6.14.04-2a3e5 6.14.04-dedca 6.16.00-23725 6.16.00-5be98 6.16.00-b4729 6.18.00-d0330 ...
For instance:
[fornarivirgo@ui01-virgo root_test]$ cat test.sub universe = vanilla Executable = test.sh ShouldTransferFiles = YES WhenToTransferOutput = ON_EXIT Log = log.log Output = log.out Error = log.err queue 1 [fornarivirgo@ui01-virgo root_test]$ cat test.sh #!/bin/bash . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_96python3 x86_64-centos7-gcc8-opt root -b -q [fornarivirgo@ui01-virgo root_test]$ condor_submit -spool -name sn-02.cr.cnaf.infn.it test.sub Submitting job(s). 1 job(s) submitted to cluster 8445482. [fornarivirgo@ui01-virgo root_test]$ condor_q -name sn-02.cr.cnaf.infn.it 8445482 -- Schedd: sn-02.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 09/10/20 17:21:42 OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS fornarivirgo ID: 8445482 9/10 17:21 _ _ 1 1 8445482.0 Total for query: 1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended Total for all users: 39956 jobs; 21240 completed, 2 removed, 13371 idle, 4218 running, 1125 held, 0 suspended [fornarivirgo@ui01-virgo root_test]$ condor_q -name sn-02.cr.cnaf.infn.it 8445482 -- Schedd: sn-02.cr.cnaf.infn.it : <131.154.192.58:9618?... @ 09/10/20 17:23:58 OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS fornarivirgo ID: 8445482 9/10 17:21 _ _ _ 1 8445482.0 Total for query: 1 jobs; 1 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended Total for all users: 39959 jobs; 21288 completed, 2 removed, 13341 idle, 4202 running, 1126 held, 0 suspended [fornarivirgo@ui01-virgo root_test]$ condor_transfer_data -name sn-02.cr.cnaf.infn.it 8445482 Fetching data files... [fornarivirgo@ui01-virgo root_test]$ cat log.err [fornarivirgo@ui01-virgo root_test]$ cat log.out ------------------------------------------------------------ | Welcome to ROOT 6.18/00 https://root.cern | | (c) 1995-2019, The ROOT Team | | Built for linuxx8664gcc on Jun 25 2019, 09:22:23 | | From tags/v6-18-00@v6-18-00 | | Try '.help', '.demo', '.license','.credits', '.quit'/'.q'| ------------------------------------------------------------