Versions Compared

Old Version 114

changes.mady.by.user Andrea Rendina

Saved on

compared with

New Version Current

changes.mady.by.user Andrea Rendina

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Other protocols which are commonly used at INFN-Tier-1 are posix, GridFTP, XrootD, WebDAV/http.

...

To transfer a file without SRM, globus-url-copy is commonly used. It is a command line program for file transfers which implements different protocols, among which gridFTP, an extension of FTP for file transfers. It supports parallel transfer streams and third-party-copy.

A personal certificate is required in order to use gridFTP. Also, the user DN has to be enabled on the gridFTP server by the sysadmin. The DN can be obtained from the certificate using the command:

Code Block
openssl x509 -noout -in $HOME/.globus/usercert.pem -subject

Then, it should be communicated to the User Support team in order to be enabled.

Before performing the actual file transfer, it is necessary to generate a proxy with the command:

Code Block
grid-proxy-init

By default, the proxy lasts 12 hours. In order to extend proxy life time, the following options can be used:

    -valid HOURS:MINUTES
    -hours HOURS

For example:

Code Block
-bash-4.2$ grid-proxy-init -hours 48
Your identity: /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
Enter GRID pass phrase for this identity:
Creating proxy ...................................... Done
Your proxy is valid until: Sun Aug 2 17:47:32 2020

After that, we can perform the transfers. This depends on the permissions and the access control list on the filesystem.
To write:

Code Block
globus-url-copy <local_path>/file gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file

To read, i.e. to get a local copy:

Code Block
globus-url-copy gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file local_copy

The <remote_path> (something like: /storage/gpfs_data/experiment) will be communicated to the user by the User Support team.
Also, the globus-url-copy command allows to do a third-party-copy of a file without getting a local copy on your own device.
This works with a simple concatenation of read and write:

Code Block
globus-url-copy gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<source_remote_path_>/file gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<destination_remote_path>/new_file

The full list of the additional options is available using:

Code Block
man globus-url-copy

Some useful options:

  • -f FILENAME : read a list of URL pairs from filename. Each line should contain sourceURL destURL. Enclose URLs with spaces in double qoutes ("). Blank lines and lines beginning with # will be ignored.
  • -df FILENAME, -dumpfile FILENAME : path to a file where untransferred URLs will be saved for later restarting. Resulting file is the same format as the -f input file. If file exists, it will be read and all other URL input will be ignored.
  • -cd, -create-dest : create destination directory if needed.
  • -r : copy files in subdirectories
  • -v, -verbose : display URLs being transferred
  • -p PARALLELISM, -parallel PARALLELISM : specify the number of parallel data connections should be used.
  • -list URL : list the files located at URL.
  • -sync : only transfer files where the destination does not exist or differs from the source. -sync-level controls how to determine if files differ.
  • -sync-level number : criteria for determining if files differ when performing a sync transfer. The default sync level is 2.
    The available levels are:
    • Level 0: will only transfer if the destination does not exist.
    • Level 1: will transfer if the size of the destination does not match the size of the source.
    • Level 2: will transfer if the time stamp of the destination is older than the time stamp of the source.
    • Level 3: will perform a checksum of the source and destination and transfer if the checksums do not match.

A user can also use the gfal tools, that are explained in the following paragraphs, for example to list the files of a directory or remove a file, respectively:

Code Block
gfal-ls gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/directory
gfal-rm gsiftp://gridftp-plain-virgo.cr.cnaf.infn.it:2811/<remote_path>/file

...

All the SRM specifications are available here [13].

In this case, a voms-proxy is needed (see in the previous sections for details on proxy generation).

Code Block
-bash-4.2$ voms-proxy-init --voms virgo:/virgo/virgo
Enter GRID pass phrase for this identity:
Contacting voms-01.pd.infn.it:15009 [/DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-01.pd.infn.it] "virgo"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_u10162.

Your proxy is valid until Tue Aug 18 22:45:07 CEST 2020
-bash-4.2$ voms-proxy-info --all
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=1185352064
issuer : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type : RFC3820 compliant impersonation proxy
strength : 1024
path : /tmp/x509up_u10162
timeleft : 11:57:53
key usage : Digital Signature, Key Encipherment
=== VO virgo extension information ===
VO : virgo
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer : /DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-01.pd.infn.it
attribute : /virgo/virgo/Role=NULL/Capability=NULL
attribute : /virgo/Role=NULL/Capability=NULL
timeleft : 11:57:53
uri : voms-01.pd.infn.it:15009

In contrast to the gridFTP protocol, we have to contact a StoRM frontend on the port 8444. Then the frontend communicates the request to a backend [14].

The Virual Organizations use dedicated storm endpoint for data managemant and data transfer:

https://www.cnaf.infn.it/~usersupport/StorageAreas.html.

For example, AMS uses storm-fe-ams for the disc area (/storage/gpfs_ams), but storm-archive to write on the buffer tape (/storage/gpfs_archive/ams).
Also, the path to read or write is not the real path on the filesystem, but all the experiments use a singular access path to the storage area.

Now we consider two tools for the SRM protocol: Gfal and ClientSRM.

...

Documentation is available here [15]. However the user can get the full list using the command man gfal-copy.

Most used commands are:

  • gfal-ls: List information about the file
  • gfal-xattr: Display attributes of a file or set them to a new value
  • gfal-cat: Concatenate file to standard output
  • gfal-mkdir: Create the DIRECTORY(ies), if they do not already exist
  • gfal-stat: Display extended information about a file or directory
  • gfal-chmod: Change the permissions of a file
  • gfal-rename: Renames SOURCE to DESTINATION
  • gfal-sum: Calculates the checksum of the specified file, using a specified checksum algoritm
  • gfal-rm: Removes each specified file or directory
  • gfal-save: Reads from stdin and writes to a file until it finds EOF

These are the steps to install Gfal assuming the machine is CentOS7:

Enable epel repo:

Code Block
curl http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm >/tmp/epel-release-latest-7.noarch.rpm
sudo rpm -ivh /tmp/epel-release-latest-7.noarch.rpm

Enable egi repo:

Code Block
echo '\[EGI-trustanchors\]name=EGI-trustanchorsbaseurl=http://repository.egi.eu/sw/production/cas/1/current/gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3gpgcheck=1enabled=1' | sudo tee /etc/yum.repos.d/egi.repo

Install several tools:

Code Block
sudo yum install -y gfal2-util gfal2-all fetch-crl ca-policy-egi-core globus-proxy-utils

Install personal certificate on the machine:

Code Block
cd $HOME
mkdir -p .globus
cd .globus
openssl pkcs12 -clcerts -nokeys -in cert.p12 -out usercert.pem
openssl pkcs12 -nocerts -in cert.p12 -out userkey.pem
chmod 600 usercert.pem
chmod 400 userkey.pem

To check all is correctly working:

Code Block
grid-proxy-init -valid 168:00
gfal-copy --version

The last command should produce a list of the available protocols. The list should include gridftp. If this is not the case, try to do: yum update.

Some examples of gfal utils below.

Code Block
-bash-4.2$ gfal-ls srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4
generated
virgo
RomePSS
virgoin
virgoout_O2LH_01_0127_067_-81.305_-90.000_.mat
prova_lucia2
prova_lucia3
vedovato
virgojob_summary_O2LH_01_0127_067_-81.305_-90.000_.mat
virgoRomePSS
testalbe
vdbroeck
prova_lucia
prova_andrea

-bash-4.2$ gfal-rm srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea DELETED

-bash-4.2$ gfal-copy /home/USER-SUPPORT/arendina/sleep.sub srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
Copying file:///home/USER-SUPPORT/arendina/sleep.sub [DONE] after 2s

-bash-4.2$ gfal-sum srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea ADLER32
srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea 2bca5372

For all the gfal commands see [16].

...

In case of local to remote transfer, you have to request the storage space in the destination filesystem and this is done with the command clientSRM PtP, where PtP stands for Prepare to Put. For example:

Code Block
$ clientSRM PtP -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea

where:

  • -e is used to specify the endpoint.
  • -s is used to specify the destination surl, which is composed by a space token (virgo4 in the example) and the file path. The space token will be communicated by the Experiment Support group.
  • -v is the verbose level, further.

and the complete list of the options is listed by the command clientSRM PtP -help or in [17].
The output should be something like this:

Code Block
-bash-4.2$ clientSRM PtP -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PtP request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
============================================================
SRM Response:
requestToken="51e58c63-afdd-4ccb-8a6d-8551b4261c33"
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
============================================================

It is important to pay attention to the request token (in this case 51e58c63-afdd-4ccb-8a6d-8551b4261c33), which will be used later. Then it is necessary to check the status of the request with clientSRM SPtP (Status of Prepare to Put) :

Code Block
clientSRM SPTP -v -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33

where with -t you provide the token shown in the output of the clientSRM PtP command. The output will show you whether the request is successful in the status field.

Code Block
-bash-4.2$ clientSRM SPTP -v -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33
============================================================
Sending StatusPtP request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All chunks successfully handled!"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_SPACE_AVAILABLE"(24)
explanation="srmPrepareToPut successfully handled!"
[0] TURL="gsiftp://ds-815.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea"
============================================================

It is important to remember the TURL which will be used in transfer command with the globus-url-copy command. After that, we can perform the file transfer:

Code Block
bash-4.2$ globus-url-copy /home/USER-SUPPORT/arendina/sleep.sub gsiftp://ds-815.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea

...

Code Block
-bash-4.2$ clientSRM Pd -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PutDone request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All file requests are successfully completed"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_SUCCESS"(0)
explanation="Success"
============================================================

Similarly, in case of remote to local transfer the command is clientSRM PtG, where PtG stands for Prepare to Get, and to check the preparation status is  clientSRM SPtG:

Code Block
-bash-4.2$ clientSRM PTG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PtG request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
============================================================
SRM Response:
requestToken="fe633fd3-de07-4a3e-a388-3cc2adf1fd3a"
arrayOfFileStatuses (size=1)
[0] status: statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
[0] sourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
============================================================

-bash-4.2$ clientSRM SPtG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t fe633fd3-de07-4a3e-a388-3cc2adf1fd3a 
============================================================
Sending StatusPtG request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All chunks successfully handled!"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0] status: statusCode="SRM_FILE_PINNED"(22)
explanation="srmPrepareToGet successfully handled!"
[0] sourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] fileSize=280
[0] transferURL="gsiftp://ds-915.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea"
============================================================

So we can perform the transfer with a globus-url-copy command:

Code Block
-bash-4.2$ globus-url-copy gsiftp://ds-915.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea copia
-bash-4.2$ ls
ce_testp308.sub copia pass sleep.sh sleep.sub test.sub

Finally, to list the file in a directory you can use the command clientSRM Ls:

Code Block
-bash-4.2$ clientSRM Ls -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4

More information on using SRM clients can be found here [18].

...

To use the XrootD protocol a voms-proxy is needed.

Code Block
-bash-4.2$ voms-proxy-init --voms ams02.cern.ch
Enter GRID pass phrase for this identity:
Contacting voms-02.pd.infn.it:15008 [/DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-02.pd.infn.it] "ams02.cern.ch"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_u25071.

Your proxy is valid until Tue Sep 08 23:12:00 CEST 2020
-bash-4.2$ voms-proxy-info --all
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=451777284
issuer : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type : RFC3820 compliant impersonation proxy
strength : 1024
path : /tmp/x509up_u25071
timeleft : 11:59:48
key usage : Digital Signature, Key Encipherment
=== VO ams02.cern.ch extension information ===
VO : ams02.cern.ch
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer : /DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-02.pd.infn.it
attribute : /ams02.cern.ch/Role=NULL/Capability=NULL
timeleft : 11:59:48
uri : voms-02.pd.infn.it:15008

Some VOs make use of dedicated xrootd endpoints which alias are in the public web page https://www.cnaf.infn.it/~usersupport/XrootD_SA.html.
In this case the user can contact directly the server or otherwise a redirector. Also, the starting part of the file system path, named local root, should be known. For instance, the server for the AMS experiment can be contacted at the 8082 port whereas 8083 is the port to contact the redirector. The local root is /storage/gpfs_ams/xrootd.

Let's see some examples of the xrd clients.

Listing directory

Code Block
-bash-4.2$ xrdfs root://xrootd-ams.cr.cnaf.infn.it:8082// ls /

/eos
/storage
/test
/test_1906
/test_1906_2
/test_1906_3
/test_2105
/test_2105_2
/test_etc_0205
-bash-4.2$ xrdfs root://xrootd-ams.cr.cnaf.infn.it:8082// ls /eos/ams
/eos/ams/Data
/eos/ams/MC
/eos/ams/cnaf-groups
/eos/ams/cnaf-users

Download

Code Block
-bash-4.2$ xrdcp root://xrootd-ams.cr.cnaf.infn.it:8082//test_1906 copia_locale
[654.6kB/654.6kB][100%][==================================================][654.6kB/s]

Upload           

Code Block
-bash-4.2$ xrdcp copia_locale root://xrootd-ams.cr.cnaf.infn.it:8082//test_0809
[654.6kB/654.6kB][100%][==================================================][654.6kB/s]

Removing a file

Code Block
-bash-4.2$ xrdfs root://xrootd-ams.cr.cnaf.infn.it:8083// rm /test_0809

...

At INFN-Tier-1, valid WebDAV endpoints for the experiments’ storage areas are provided with StoRM WebDAV (third-party-copy supported) or Apache.

Then, the most common WebDAV clients can be used to access the storage areas, namely browsers and command-line tools such as curl and gfal.

When StoRM WebDAV is used, VOMS proxies are supported only by command-line tool, and browsers can be used to navigate into the storage area content if anonymous read-only access is enabled (HTTP endpoint) or if VO users access by their X509 certificate is enabled (HTTPS endpoint).

A few useful commands follow and more info are available at [19].

With a valid voms-proxy:

Code Block
[arendina@ui-tier1 ~]$ voms-proxy-init --voms juno
Contacting lcgvoms02.jinr.ru:15008 [/C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru] "juno"...
Remote VOMS server contacted succesfully.
Created proxy in /tmp/x509up_u10162.
Your proxy is valid until Fri Jul 02 05:42:21 CEST 2021

[arendina@ui-tier1 ~]$ voms-proxy-info --all
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=2090475310
issuer    : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity  : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type      : RFC3820 compliant impersonation proxy
strength  : 2048
path      : /tmp/x509up_u10162
timeleft  : 11:59:35
key usage : Digital Signature, Key Encipherment
=== VO juno extension information ===
VO        : juno
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer    : /C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru
attribute : /juno/Role=NULL/Capability=NULL
timeleft  : 11:59:35
uri       : lcgvoms02.jinr.ru:15008

Listing directory

Code Block
[arendina@ui-tier1 ~]$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno
rucio4juno
cronTest

Upload

Code Block
[arendina@ui-tier1 ~]$ gfal-copy test.txt davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107
Copying file:///home/USER-SUPPORT/arendina/test.txt   [DONE]  after 0s

Download             

Code Block
[arendina@ui-tier1 ~]$ gfal-copy davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 here
Copying davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107   [DONE]  after 0s

Removing a file

Code Block
[arendina@ui-tier1 ~]$ gfal-rm davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107
davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107  DELETED

...

In order to properly perform a third-party-copy between two endpoints which support the http protocol it occurs to set the BEARER_TOKEN up, also known as macaroon.
Indeed, this token is used to authenticate the user always to the second endpoint. For this reason, the server to contact to get the token depends on whether the copy is in pull or push mode.

Actually, if both the endpoints are able to release a macaroon and the used gfal version is greater or equal to the 2.15 [28], the process is automatic.
Otherwise, if only one of the two endpoints can release a BEARER_TOKEN, or equivantly just one endpoint supports the http protocol, the user must request a macaroon to that endpoint.

Two easy examples follow below.

Pull-copy

The user has to authenticate itself with a valid voms-proxy to the first endpoint, in this case IHEP, and then IHEP will authenticate itself with a BEARER_TOKEN that the user has got previously from the second endpoint, namely the CNAF server (this happens because a pull-copy triggers a GET request). So, step by step:

I ask a BEARER_TOKEN to the second endpoint, namely "xfer-archive" at CNAF.

Code Block
-bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

I can pull a file from CNAF to IHEP.

Code Block
-bash-4.2$ gfal-copy -v --copy-mode pull davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_Andrea davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big

Push-copy

On the other hand, since the push-copy triggers a COPY request, the second endpoint to contact is CNAF again.

I ask a BEARER_TOKEN to the second endpoint, as above.

Code Block
-bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

Then, I can push a file from IHEP to CNAF.

Code Block
-bash-4.2$ gfal-copy -v --copy-mode push davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_to_CNAF

...

StoRM WebDAV also supports OpenID connect authentication and authorization on storage areas, so tokens can be used instead of proxies [20]. Dedicated IAM (Identity and Access Management) instances can be configured for the experiments upon requests (please contact the user support).

Also, a catch-all IAM instance is available at iam-t1-computing.cloud.cnaf.infn.it where specific groups can be created for different communities, and users can register and request group membership. In this case, exploiting the support for fine-grained authorization within StoRM WebDAV, users can browse their data after OIDC login, and/or they can access storage from command line using gfal (Gfal-utils) or curl with JWT tokens.

Once registered within IAM, an access token can be retrieved using OIDC-agent, which is available in ui-tier1. A full set of commands follows below.

At first, we have to start the process with the command

Code Block
[arendina@ui-tier1 ~]$ eval `oidc-agent-service use`
Agent pid 20578

Then, one needs to register his own client (this has to be done just the first time to create a new account):

Code Block
[arendina@ui-tier1 ~]$ oidc-gen

following the steps required by the application, and namely:

  • initialize his OIDC client's name
  • insert IAM URL (e.g. https://iam-t1-computing.cloud.cnaf.infn.it/)
  • when a web page for authentication opens, enter the credentials of the IAM account
  • authorize the new client
  • insert your client's password twice.

Whereas, from the second time it is enough typing:

Code Block
[arendina@ui-tier1 ~]$ oidc-add <your-client-name>
Enter decryption password for account config '<your-client-name>':
success

To get the access token and save it an environment variable:

Code Block
[arendina@ui-tier1 ~]$ TOKEN=$(oidc-token <your-client-name>)

The token will be valid for 60 minutes. It can be obtained a new token issuing the oidc-token command again.
At the end, stop oidc-agent daemon:

Code Block
oidc-agent-service stop

...

Code Block
-bash-4.2$ eval `oidc-agent-service use`
Agent pid 17216
-bash-4.2$ oidc-add juno2
Enter decryption password for account config 'juno2':
success
-bash-4.2$ TOKEN=$(oidc-token juno2)
-bash-4.2$ export BEARER_TOKEN=$TOKEN
-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno/
rucio4juno
test_Andrea
cronTest

NB: an error like this could appear, but it is just an irrelevent warning message:

Code Block
(Davix::OpenSSL) Error: impossible to open /tmp/x509up_u10164:  : error:02001002:system library:fopen:No such file or directory

On the other hand, to switch back to the voms-proxy it occurs to unset the BEARER_TOKEN, namely:

Code Block
-bash-4.2$ unset BEARER_TOKEN

-bash-4.2$ voms-proxy-init --voms belle
Contacting voms.cc.kek.jp:15020 [/C=JP/O=KEK/OU=CRC/CN=host/voms.cc.kek.jp] "belle"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_ucopy test https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/test_now

-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/belle
bellehttpd
TMP
CONTENT.stats
rucio4iddls

...

Code Block
[fornariclas12@ui-tier1 ~]$ eval `oidc-agent-service use`
Agent pid 24806

[fornariclas12@ui-tier1 ~]$ oidc-gen
Enter short name for the account to configure: fornariclas12
[1] https://iam-test.indigo-datacloud.eu/
[2] https://iam.deep-hybrid-datacloud.eu/
[3] https://iam.extreme-datacloud.eu/
[4] https://b2access.eudat.eu/oauth2/
[5] https://b2access-integration.fz-juelich.de/oauth2
[6] https://unity.eudat-aai.fz-juelich.de/oauth2/
[7] https://unity.helmholtz-data-federation.de/oauth2/
[8] https://login.helmholtz-data-federation.de/oauth2/
[9] https://services.humanbrainproject.eu/oidc/
[10] https://accounts.google.com/
[11] https://aai.egi.eu/oidc/
[12] https://aai-dev.egi.eu/oidc
[13] https://login.elixir-czech.org/oidc/
[14] https://oidc.scc.kit.edu/auth/realms/kit/
Issuer [https://iam-test.indigo-datacloud.eu/]: https://iam-t1-computing.cloud.cnaf.infn.it/     
Space delimited list of scopes or 'max' [openid profile offline_access]: 
Registering Client ...
Generating account configuration ...
accepted
To continue and approve the registered client visit the following URL in a Browser of your choice:
https://iam-t1-computing.cloud.cnaf.infn.it/authorize?response_type=code&client_id=d70f9284-c371-4ce2-90e9-1cd3d6201329&redirect_uri=http://localhost:13481&scope=openid profile offline_access&access_type=offline&prompt=consent&state=0:2s4EBiNy_Z0a835JBiaw_cGG:33:L3RtcC9vaWRjLTRzMjdBOS9vaWRjLWFnZW50LjExMTY2&code_challenge_method=S256&code_challenge=cL1sKfZAgVO7rV5PGm5XXFKgPiM-Dbnj_Bf3BpzEoR4
/bin/xdg-open: line 881: www-browser: command not found
/bin/xdg-open: line 881: links2: command not found
/bin/xdg-open: line 881: elinks: command not found
/bin/xdg-open: line 881: links: command not found
Polling oidc-agent to get the generated account configuration ...success
The generated account config was successfully added to oidc-agent. You don't have to run oidc-add.
Enter encryption password for account configuration 'fornariclas12': 
Confirm encryption Password: 

[fornariclas12@ui-tier1 ~]$ oidc-token fornariclas12
eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJkODFhNTQ5OC01ODlhLTQwYjItODJhMi04M2U2MjY3NTQwMDQiLCJpc3MiOiJodHRwczpcL1wvaWFtLXQxLWNvbXB1dGluZy5jbG91ZC5jbmFmLmluZm4uaXRcLyIsIm5hbWUiOiJGZWRlcmljbyBGb3JuYXJpIiwiZ3JvdXBzIjpbImZhemlhIiwiamxhYjEyIl0sInByZWZlcnJlZF91c2VybmFtZSI6ImZmb3JuYXJpIiwib3JnYW5pc2F0aW9uX25hbWUiOiJ0MS1jb21wdXRpbmciLCJleHAiOjE2MjA5MDYxNDksImlhdCI6MTYyMDkwMjU0OSwianRpIjoiNjM0Y2VjNzMtMDAwMC00MGU0LTg5YjgtZjI2NjFjMGZiZmE3IiwiY2xpZW50X2lkIjoiZDcwZjkyODQtYzM3MS00Y2UyLTkwZTktMWNkM2Q2MjAxMzI5In0.irwbiCxdHZWOpyv1l9jbmGExeVl_gyB9fhHEnlSjbFOL4WV--vrh3E3pLu-e1FV8c23TexwbHvnJGEZxaOqVNJMO3VUaso2i0zIz2oPTsEmxJpa52goNTYrbRm2TRGErkAE0vaEqJbbI3wDoCJ_BPkdDLAT0YL5-AbHTkum8Ovs

[fornariclas12@ui-tier1 ~]$ export AT=$(oidc-token fornariclas12)

[fornariclas12@ui-tier1 ~]$ cat test.sub
universe                = vanilla
executable              = test.sh
getenv			        = true
log                     = test.log
output                  = outfile.txt
error                   = errors.txt
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
queue

[fornariclas12@ui-tier1 ~]$ cat test.sh
#!/bin/bash
echo "this is a test by F. Fornari" | tee test-fornari
export BEARER_TOKEN=$AT
gfal-copy test-fornari https://xfer-archive.cr.cnaf.infn.it:8443/jlab12/test-fornari
gfal-ls https://xfer-archive.cr.cnaf.infn.it:8443/jlab12
ls -lrth /storage/gpfs_data/class12vo/data/
gfal-rm https://xfer-archive.cr.cnaf.infn.it:8443/jlab12/test-fornari

[fornariclas12@ui-tier1 ~]$ condor_submit -spool -name sn-02 test.sub
Submitting job(s).
1 job(s) submitted to cluster 15750247.

[fornariclas12@ui-tier1 ~]$ watch condor_q -name sn-02 15750247

[fornariclas12@ui-tier1 ~]$ condor_transfer_data -name sn-02 15750247
Fetching data files...

[fornariclas12@ui-tier1 ~]$ cat outfile.txt 
this is a test by F. Fornari
test-fornari
total 512
-rw-r-----+ 1 storm storm 29 May 13 15:29 test-fornari
[fornariclas12@ui-tier1 ~]$ cat errors.txt 
[fornariclas12@ui-tier1 ~]$ cat test.log 
000 (15750247.000.000) 05/13 15:28:44 Job submitted from host: <131.154.192.58:9618?addrs=131.154.192.58-9618&noUDP&sock=703648_945d_3>
...
001 (15750247.000.000) 05/13 15:29:29 Job executing on host: <131.154.208.195:9618?addrs=131.154.208.195-9618&noUDP&sock=14671_aa08_3>
...
006 (15750247.000.000) 05/13 15:29:30 Image size of job updated: 237380
	0  -  MemoryUsage of job (MB)
	0  -  ResidentSetSize of job (KB)
...
005 (15750247.000.000) 05/13 15:29:30 Job terminated.
	(1) Normal termination (return value 0)
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Total Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Total Local Usage
	136  -  Run Bytes Sent By Job
	544  -  Run Bytes Received By Job
	136  -  Total Bytes Sent By Job
	544  -  Total Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated 
	   Cpus                 :     0.02        1         1 
	   Disk (KB)            :    25           1   3400414 
	   Memory (MB)          :     0           1       128 
...

[fornariclas12@ui-tier1 ~]$ oidc-agent-service stop
unset OIDC_SOCK;
unset OIDCD_PID;
echo Agent pid 24806 killed;
unset OIDCD_PID_FILE;

...

Data on tape need to be copied on a disk buffer to be accessed.
The buffer is a disk (detached and generally different from the actual disk) that serves as a temporary platform for files that must be migrated or have been recalled from tape. This is not a static disk but once it is full, the oldest and already migrated files are deleted by the garbage collector (however the files remain on tape).
Tape area path will be provided by CNAF. In particular, you can check the list of storage areas at the following page: https://www.cnaf.infn.it/~usersupport/StorageAreas.html where you can find the correspondence between the storage area and the real path on the filesystem.

...

To know if a file is on the disk, when the user is provided with POSIX access, one can check the file dimension using a command like this:

Code Block
-bash-4.2$ ls -ls /storage/gpfs_archive/argo/argo-cnaf/testkaren/provaka1_1.txt
0 -rw-r----- 1 argoprod argo 27 Dec 15 2010 /storage/gpfs_archive/argo/argo-cnaf/testkaren/provaka1_1.txt

If the file has null dimension (field number 1 of the output), it is not physically present on the disk, whereas if the actual size (field number 6) is not 0, the file is on tape.

...

To check if the file is on disk, on tape or both, a user can use the following command (to be used with a valid VOMS Proxy):

Code Block
clientSRM ls -l -v NIG -e <endpoint> -s <file-SURL>

For example:

Code Block
clientSRM ls -l -v NIG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444/ -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/ams/${your_file}

Based on the information shown in the output, it is possible to locate the file by checking the value of the fileLocality line:

-[0] fileLocality=0         the the file is on disk

-[0] fileLocality=1         the file is on tape

-[0] fileLocality=2         the file is both on disk and tape 

...

As previously explained, when a file has to be moved to tape, one needs to put it into the buffer disk. From there, data will be automatically migrated to tape after a certain time.

The user can write a file on the buffer disk with a command like this:

Code Block
gfal-copy your_file srm://storm-fe-archive.cr.cnaf.infn.it:8444/ams/your_file

The file "your_file" will be automatically migrated to tape. To check if the file has been migrated, a user can use a command like the following:

Code Block
clientSRM ls -l -v NIG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444/ -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/ams/your_file

If the output contains a line like this:

[0] fileLocality=0

the file has not been migrated yet. If the output contains a line like this:

[0] fileLocality=2

the file has already been migrated to tape.

...

If a user needs a file that is only on tape, it has to be recalled back to disk. This can be done directly with a clientSRM bol, where "bol" stands for Bring On Line, with a command like this:

Code Block
clientSRM bol -e httpg://storm-test.cr.cnaf.infn.it:8444 -s srm://storm-test.cr.cnaf.infn.it:8444/tape/file_test

then the output returns a requestToken (for example: requestToken="ea8b525d-1b12-47a5-b8d5-6935ebc53003"), which it can be used to check the status of the Bring On Line operation:

Code Block
clientSRM sbol -e httpg://storm-test.cr.cnaf.infn.it:8444 -t "ea8b525d-1b12-47a5-b8d5-6935ebc53003"

When the operation is completed, the user gets a Site URL (SURL), that one can exploit to access the recalled file. Whereas, If a user needs to recall multiple files, it can be issued a multiSURL clientSRM bol command:

Code Block
clientSRM bol -e httpg://storm-test.cr.cnaf.infn.it:8444 -s srm://storm-test.cr.cnaf.infn.it:8444/tape/file_test1 srm://storm-test.cr.cnaf.infn.it:8444/tape/file_test2

After being recalled, the files get pinned. If the user is provided with POSIX access to the filesystem, they can check extended attributes of a file, for example:

Code Block
getfattr -d /storage/gemss_test1/tape/file_test
# file: storage/gemss_test1/tape/file_test
user.TSMRecD="1606379599"
user.TSMRecR="0"
user.storm.migrated
user.storm.pinned="1606638797"

Default pin time is 3 days on storm-test, configurable by StoRM admins. After that time, the file could be removed from disk buffer. It can be specified a custom pin lifetime in seconds with -b option of the clientSRM bol command, for example:

Code Block
clientSRM bol -b 86400 -e httpg://storm-test.cr.cnaf.infn.it:8444 -s srm://storm-test.cr.cnaf.infn.it:8444/tape/file_test

so, this time pin lifetime will be 1 day only. In any case, the user can check extended attributes of the file with gfal-xattr:

Code Block
gfal-xattr srm://storm-test.cr.cnaf.infn.it:8444/tape/file_test
user.replicas = gsiftp://transfer-test.cr.cnaf.infn.it:2811//storage/gemss_test1/tape/file_test
user.status = ONLINE_AND_NEARLINE
srm.type = StoRM

The user can see that the recalled file is ONLINE_AND_NEARLINE, i.e. it's both on disk and tape. Alternatively to clientSRM bol and clientSRM sbol, they can use, respectively, clientSRM PtG and clientSRM SPtG.

NB: gfal-xattr ALWAYS recall a file. This is a bug of the command that will be fixed by the gfal utils developers.   

...

To recall files from tape, it is necessary to provide the list of the file to be recalled. CNAF will recall them.
Please, contact the User Support team at the mailing list user-support@lists.cnaf.infn.it.