General informations
HTCondor is a powerful yet complex batch system: in this page you'll find basic informations on how to submit and manage jobs, but for a fully fledged user manual you can find all the details on the official HTCondor documentation.
All the command you will find on the user guide need to be executed on the frontend node, the one you'll land via ssh (htc-fe.rmlab.infn.it)
Submit a job
$ condor_submit example.sub
...
The .sub file is called "job descriptor" where you can write all the informations related to the job you want to submit (universuniverse, job numbers, parameters, etc)
...
* Executable = foo.sh
* Universe = vanilla
Error = bar/err.$(ClusterId).$(ProcId)
Output = bar/out.$(ClusterId).$(ProcId)
* Log = bar/log.$(ClusterId).$(ProcId)
arguments = arg1 arg2 $(ProcId)
request_cpus = 1
* request_memory = 1024
request_disk = 10240
should should_transfer_files = yesno
* Queue 1
* = required fields
You can also submit many similar jobs with one queue command
For reference you can have a look at job submission manual or at science job example
...
Parametri:
Executable
...
=
...
Job
...
to
...
be
...
executed
Universe
...
=
...
choose
...
between
...
vanilla/docker/parallel
Error,
...
Output,
...
Log
...
=
...
where
...
to
...
write
...
log
...
files.
...
Those
...
folders
...
need
...
to
...
exist
...
before
...
submitting
...
the
...
job
Arguments
...
=
...
Strings
...
array
...
to
...
be
...
provided
...
to
...
the
...
job
...
itself.
request_[...]
...
=
...
Job
...
requirements
Queue
...
=
...
This
...
is
...
the
...
submission
...
request,
...
here
...
you
...
can
...
specify
...
the
...
number of job to be executed
Require a specific operative system environment
To provide the highest compatibility with scientific frameworks, the cluster is composed of heterogeneous nodes running Ubuntu 24 and AlmaLinux 9.
Unless explicitly requested, jobs may be executed on any available node — this is useful for jobs that use self-contained environments, such as Python virtual environments or Docker/Singularity containers.
To explicitly request that a job run on a specific operating system, simply add the following line to your job file:
Requirements = (OpSysName == "<OS Name>")The allowed values for <OS Name> are "AlmaLinux" and "Ubuntu".
Universe
A universe represent on an execution environment for the job: HTCondor supporto support several universes :
* vanilla
grid
java
scheduler
local
* parallel
vm
container
* docker
...
Further details about docker universe applications can be found on the official manual
Parallel Universe
Parallel universe allows you to submit parallel jobs using mpi: the scheduler will wait for all the needed resources to be available before submitting the job.
universe = parallel
executable = foo.sh
log = logfile
input = infile.$(Node)
output = outfile.$(Node)
error = errfile.$(Node)
machine_count = 2
request_cpus = 8
queue
In caso si voglia utilizzare Openmpi è necessario utilizzare uno script per permettere la comunicazione tra i vari jobs.
Tale script si trova preconfigurato sul nodo submitter nella cartella /OpenmpiScript del nodo submitter (192.168.0.102). Copiare il file .sh nella stessa folder del file di sottomissione.
Le seguenti direttive indicano rispettivamente quanti processi paralleli lanciare e quante cpu assegnare per processo:
...
Nel file di sottomissione è necessario quindi specificare come eseguibile lo script preconfigurato e come argomento il processo buildato mpi.
...
To use OpenMPI you need to use a wrapper script instead of your executable: the executable, compiled with openmpi, needs to be passed as first argument on transferred using the "trasfer_input_file" directive.
universe = parallel
executable = openmpiscript.sh
arguments = mpitest
should_transfer_files
...
=
...
yes
transfer_input_files
...
=
...
mpitest
when_to_transfer_output
...
=
...
on_exit_or_evict
output
...
=
...
logs/out.$(NODE)
error
...
...
=
...
logs/err.$(NODE)
log
...
...
=
...
logs/log
machine_count = 15
request_cpus = 10
queue
The openmpiscript.sh is locate in the /scripts folder of the frontend node, it needs to be copied in the folder where you submit the job
The following directives indicate how many parallel processes need to be run and how many cpus are needed for each process.
machine_count = 15
request_cpus = 10queue
Detailed info about parallel applications can be found here
Jobs with GPU
To execute a job that needs a GPU you can specify "request_GPUs = 1" in the job submission file.Trovate maggiori informazioni qui:
https://htcondor.readthedocs.io/en/latest/users-manual/parallel-applications.html
Manage a job
$ condor_rm clusterID.procID
...
$ condor_hold clusterID.procID
Put a job on hold, it will not be executed until released
$ condor_release clusterID.procID
Release a previously hold job.
Queue check
$ condor_q <parameters>
The condor_q parameters allows you to check a job / queue status
Parameters:
-nobatch Display a job per line
-global Shows all the active queues in the cluster
-hold Shows information about job on hold (you can also use -analyze or --better-analyze parameters)
-run Shows info on running job, to be used with -nobatch
...