To transfer a file without SRM, globus-url-copy is commonly used. It is a command line program for file transfers which implements different protocols, among which gridFTP, an extension of FTP for file transfers. It supports parallel transfer streams and third-party-copy.

A personal certificate is required in order to use gridFTP. Also, the user DN has to be enabled on the gridFTP server by the sysadmin. The DN can be obtained from the certificate using the command:

openssl x509 -noout -in $HOME/.globus/usercert.pem -subject

Then, it should be communicated to the User Support team in order to be enabled.

Before performing the actual file transfer, it is necessary to generate a proxy with the command:

grid-proxy-init

By default, the proxy lasts 12 hours. In order to extend proxy life time, the following options can be used:

    -valid HOURS:MINUTES
    -hours HOURS

For example:

-bash-4.2$ grid-proxy-init -hours 48
Your identity: /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
Enter GRID pass phrase for this identity:
Creating proxy ...................................... Done
Your proxy is valid until: Sun Aug 2 17:47:32 2020

After that, we can perform the transfers. This depends on the permissions and the access control list on the filesystem.
To write:

globus-url-copy <local_path>/file gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file

To read, i.e. to get a local copy:

globus-url-copy gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file local_copy

The <remote_path> (something like: /storage/gpfs_data/experiment) will be communicated to the user by the User Support team.
Also, the globus-url-copy command allows to do a third-party-copy of a file without getting a local copy on your own device.
This works with a simple concatenation of read and write:

globus-url-copy gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<source_remote_path_>/file gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<destination_remote_path>/new_file

The full list of the additional options is available using:

man globus-url-copy

Some useful options:

  • -f FILENAME : read a list of URL pairs from filename. Each line should contain sourceURL destURL. Enclose URLs with spaces in double qoutes ("). Blank lines and lines beginning with # will be ignored.
  • -df FILENAME, -dumpfile FILENAME : path to a file where untransferred URLs will be saved for later restarting. Resulting file is the same format as the -f input file. If file exists, it will be read and all other URL input will be ignored.
  • -cd, -create-dest : create destination directory if needed.
  • -r : copy files in subdirectories
  • -v, -verbose : display URLs being transferred
  • -p PARALLELISM, -parallel PARALLELISM : specify the number of parallel data connections should be used.
  • -list URL : list the files located at URL.
  • -sync : only transfer files where the destination does not exist or differs from the source. -sync-level controls how to determine if files differ.
  • -sync-level number : criteria for determining if files differ when performing a sync transfer. The default sync level is 2.
    The available levels are:
    • Level 0: will only transfer if the destination does not exist.
    • Level 1: will transfer if the size of the destination does not match the size of the source.
    • Level 2: will transfer if the time stamp of the destination is older than the time stamp of the source.
    • Level 3: will perform a checksum of the source and destination and transfer if the checksums do not match.

A user can also use the gfal tools, that are explained in the following paragraphs, for example to list the files of a directory or remove a file, respectively:

gfal-ls gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/directory
gfal-rm gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file
  • No labels