...
Some helpful examples follow below.
| Anchor | ||||
|---|---|---|---|---|
|
HTCondor allows multiple job submission by using the queue command.
For jobs which don't depend on parameters, it is possible to submit the same job many times specifying queue <N> in the submission file, where <N> is an integer number.
Here's a .sub file example to submit a simple job for 3 times:
| Code Block |
|---|
-bash-4.2$ cat sleep.sub
# Unix submit description file
# sleep.sub -- simple sleep job
executable = sleep.sh
log = sleep.log
output = outfile$(Process).txt
error = errors$(Process).txt
should_transfer_files = Yes
when_to_transfer_output = ON_EXIT
queue 3 |
And then run the usual commands:
| Code Block |
|---|
-bash-4.2$ condor_submit -name sn-02.cr.cnaf.infn.it -spool sleep.sub
Submitting job(s)...
3 job(s) submitted to cluster 4588631.
-bash-4.2$
-bash-4.2$ condor_q -name sn-02.cr.cnaf.infn.it
-- Schedd: sn-02.cr.cnaf.infn.it : <131.154.192.42:9618?... @ 11/04/22 11:16:11
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
dlattanziobelle ID: 4588631 11/4 11:12 _ _ _ 3 4588631.0-2
Total for query: 3 jobs; 3 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for dlattanziobelle: 3 jobs; 3 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for all users: 38425 jobs; 30994 completed, 0 removed, 636 idle, 6613 running, 182 held, 0 suspended
-bash-4.2$ condor_transfer_data -name sn-02.cr.cnaf.infn.it 4588631
Fetching data files... |
On the other hand, if the jobs depend on a parameter, it is possible to provide the queue command with a list of items, for instance the list of the files that the jobs depend on.
In this case, in the submission file it is possible to define a variable (e.g. file), which can be recalled, for instance, in arguments = $(file) and then expand such variable into a list of values (it might also be a list of lists). It is possible to express each item either in a comma and/or space separated list, either by placing each of them on different lines and delimiting the list with parentheses. It is required to specify the keyword from in the queue command. For example:
| Code Block |
|---|
executable = ...
arguments = $(file)
...
queue file from (
/storage/gpfs_data/.../file1
/storage/gpfs_data/.../file2
/storage/gpfs_data/.../fileN
) |
Another way consists to compose the list with a rule and then using the keyword matching to match a specific expression. In the following example, assuming to have a set of .root files, HTCondor will submit a job for each file matching the specified rule:
| Code Block |
|---|
executable = ...
arguments = $(file)
...
queue file matching files /storage/gpfs_data/.../*.root |
Further details are available in the official HTCondor Manual [34].
| Anchor | ||||
|---|---|---|---|---|
|
...