Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In this case, a voms-proxy is needed (see in the previous sections for details on proxy generation).

Code Block
languagebash
themeMidnight
-bash-4.2$ voms-proxy-init --voms virgo:/virgo/virgo
Enter GRID pass phrase for this identity:
Contacting voms-01.pd.infn.it:15009 [/DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-01.pd.infn.it] "virgo"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_u10162.

Your proxy is valid until Tue Aug 18 22:45:07 CEST 2020
-bash-4.2$ voms-proxy-info --all
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=1185352064
issuer : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type : RFC3820 compliant impersonation proxy
strength : 1024
path : /tmp/x509up_u10162
timeleft : 11:57:53
key usage : Digital Signature, Key Encipherment
=== VO virgo extension information ===
VO : virgo
subject : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer : /DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-01.pd.infn.it
attribute : /virgo/virgo/Role=NULL/Capability=NULL
attribute : /virgo/Role=NULL/Capability=NULL
timeleft : 11:57:53
uri : voms-01.pd.infn.it:15009

...

Anchor
Gfal utils
Gfal utils
Gfal utils

Documentation is available here [15]. However the user can get the full list using the command man gfal-copy.

...

  • gfal-ls: List information about the file
  • gfal-xattr: Display attributes of a file or set them to a new value
  • gfal-cat: Concatenate file to standard output
  • gfal-mkdir: Create the DIRECTORY(ies), if they do not already exist
  • gfal-stat: Display extended information about a file or directory
  • gfal-chmod: Change the permissions of a file
  • gfal-rename: Renames SOURCE to DESTINATION
  • gfal-sum: Calculates the checksum of the specified file, using a specified checksum algoritm
  • gfal-rm: Removes each specified file or directory (using the option -r, -R, --recursive:  remove directories and their contents recursively)
  • gfal-save: Reads from stdin and writes to a file until it finds EOF

...

  1. Enable epel repo:

    Code Block
    languagebash
    themeMidnight
    curl http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm >/tmp/epel-release-latest-7.noarch.rpm
    sudo rpm -ivh /tmp/epel-release-latest-7.noarch.rpm


  2. Enable egi repo:

    Code Block
    languagebash
    themeMidnight
    echo '\[EGI-trustanchors\]name=EGI-trustanchorsbaseurl=http://repository.egi.eu/sw/production/cas/1/current/gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3gpgcheck=1enabled=1' | sudo tee /etc/yum.repos.d/egi.repo


  3. Install several tools:

    Code Block
    languagebash
    themeMidnight
    sudo yum install -y gfal2-util gfal2-all fetch-crl ca-policy-egi-core globus-proxy-utils


    Install personal certificate on the machine:

    Code Block
    languagebash
    themeMidnight
    cd $HOME
    mkdir -p .globus
    cd .globus
    openssl pkcs12 -clcerts -nokeys -in cert.p12 -out usercert.pem
    openssl pkcs12 -nocerts -in cert.p12 -out userkey.pem
    chmod 600 usercert.pem
    chmod 400 userkey.pem


To check all is correctly working:

Code Block
languagebash
themeMidnight
grid-proxy-init -valid 168:00
gfal-copy --version

...

Some examples of gfal utils below.

Code Block
languagebash
themeMidnight
-bash--bash-4.2$ gfal-ls srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4
generated
virgo
RomePSS
virgoin
virgoout_O2LH_01_0127_067_-81.305_-90.000_.mat
prova_lucia2
prova_lucia3
vedovato
virgojob_summary_O2LH_01_0127_067_-81.305_-90.000_.mat
virgoRomePSS
testalbe
vdbroeck
prova_lucia
prova_andrea

-bash-4.2$ gfal-rm srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea DELETED

-bash-4.2$ gfal-copyrm /home/USER-SUPPORT/arendina/sleep.sub-R srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4ams/prova_andrea
Copying file:///home/USER-SUPPORT/arendina/sleep.sub [DONE] after 2s

-bash-4.2$ gfal-sum folder_test
srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4ams/prova_andrea ADLER32
folder_test/dlattanzio2ams.txt  DELETED
srm://storm-fe-archive.cr.cnaf.infn.it:8444/ams/folder_test     RMDIR

-bash-4.2$ gfal-copy /home/USER-SUPPORT/arendina/sleep.sub srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea 2bca5372

For all the gfal commands see [16].

...

In case of local to remote transfer, you have to request the storage space in the destination filesystem and this is done with the command clientSRM PtP, where PtP stands for Prepare to Put. For example:

Code Block
$ clientSRM PtP -e httpg
Copying file:///home/USER-SUPPORT/arendina/sleep.sub [DONE] after 2s

-bash-4.2$ gfal-sum srm://storm-fe-archive.cr.cnaf.infn.it:8444 -s /virgo4/prova_andrea ADLER32
srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea

where:

  • -e is used to specify the endpoint.
  • -s is used to specify the destination surl, which is composed by a space token (virgo4 in the example) and the file path. The space token will be communicated by the Experiment Support group.
  • -v is the verbose level, further.

and the complete list of the options is listed by the command clientSRM PtP -help or in [17].
The output should be something like this:

 2bca5372

For some experiments it is also possibile to use the https or davs protocol contacting directly the StoRM frontend in the same way, but some configuration changes of the gfal tool are needed on the client. In fact, the SRM plugin specifcs into the /etc/gfal2.d/srm_plugin.conf file can be ordered as in the following example:

Code Block
languagebash
themeMidnight
# ordered list of turls protocols for remote I/O
# the top priority protocol is the first one
TURL_PROTOCOLS=davs;gsiftp;rfio;gsidcap;dcap;kdcap

# ordered list of turls protocols for third party transfer
# only protocol supporting third party copy should be here
# the top priority protocol is the first one
TURL_3RD_PARTY_PROTOCOLS=https;gsiftp;root

For all the gfal commands see [16].

Anchor
ClientSRM utils
ClientSRM utils
ClientSRM utils

In case of local to remote transfer, you have to request the storage space in the destination filesystem and this is done with the command clientSRM PtP, where PtP stands for Prepare to Put. For example:

Code Block
languagebash
themeMidnight
$ clientSRM PtP -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea

where:

  • -e is used to specify the endpoint.
  • -s is used to specify the destination surl, which is composed by a space token (virgo4 in the example) and the file path. The space token will be communicated by the Experiment Support group.
  • -v is the verbose level, further.

and the complete list of the options is listed by the command clientSRM PtP -help or in [17].
The output should be something like this:

Code Block
languagebash
themeMidnight
-bash-4.2$ clientSRM PtP -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PtP request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
============================================================
SRM Response:
requestToken="51e58c63-afdd-4ccb-8a6d-8551b4261c33"
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
============================================================

It is important to pay attention to the request token (in this case 51e58c63-afdd-4ccb-8a6d-8551b4261c33), which will be used later. Then it is necessary to check the status of the request with clientSRM SPtP (Status of Prepare to Put) :

Code Block
languagebash
themeMidnight
clientSRM SPTP -v -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33

where with -t you provide the token shown in the output of the clientSRM PtP command. The output will show you whether the request is successful in the status field.

Code Block
languagebash
themeMidnight
-bash-4.2$ clientSRM SPTP -v -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33
============================================================
Sending StatusPtP request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All chunks successfully handled!"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_SPACE_AVAILABLE"(24)
explanation="srmPrepareToPut successfully handled!"
[0] TURL="gsiftp://ds-815.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea"
============================================================

It is important to remember the TURL which will be used in transfer command with the globus-url-copy command. After that, we can perform the file transfer:

Code Block
languagebash
themeMidnight
bash-4.2$ globus-url-copy /home/USER-SUPPORT/arendina/sleep.sub gsiftp://ds-815.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea

Actually, with this command we overwrite the file "prova_andrea", prepared with the clientSRM PtP command, with the local file "sleep.sub".

At the end, in order to avoid the open request in the StoRM database, we have to finish with the clientSRM Pd command where Pd stands for Put done:

Code Block
languagebash
themeMidnight
-bash-4.2$ clientSRM Pd -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PutDone request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All file requests are successfully completed"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_SUCCESS"(0)
explanation="Success"
============================================================

Similarly, in case of remote to local transfer the command is clientSRM PtG, where PtG stands for Prepare to Get, and to check the preparation status is  clientSRM SPtG:

Code Block
languagebash
themeMidnight
-bash-4.2$ clientSRM PTG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
===================
Code Block
-bash-4.2$ clientSRM PtP -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PtP request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
Sending PtG request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
SRM Response:
requestToken="51e58c63-afdd-4ccb-8a6d-8551b4261c33"
arrayOfFileStatuses (size=1)
[0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
============================================================

It is important to pay attention to the request token (in this case 51e58c63-afdd-4ccb-8a6d-8551b4261c33), which will be used later. Then it is necessary to check the status of the request with clientSRM SPtP (Status of Prepare to Put) :

Code Block
clientSRM SPTP -v -e httpg
SRM Response:
requestToken="fe633fd3-de07-4a3e-a388-3cc2adf1fd3a"
arrayOfFileStatuses (size=1)
[0] status: statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
[0] sourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33

where with -t you provide the token shown in the output of the clientSRM PtP command. The output will show you whether the request is successful in the status field.

Code Block
.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
============================================================

-bash-4.2$ clientSRM SPTPSPtG -v -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63fe633fd3-afddde07-4ccb4a3e-8a6da388-8551b4261c333cc2adf1fd3a 
============================================================
Sending StatusPtPStatusPtG request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All chunks successfully handled!"
============================================================
SRM Response:
arrayOfFileStatuses (size=1)=1)
[0] status: statusCode="SRM_FILE_PINNED"(22)
explanation="srmPrepareToGet successfully handled!"
[0] SURLsourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] status: statusCode="SRM_SPACE_AVAILABLE"(24)
explanation="srmPrepareToPut successfully handled!"
fileSize=280
[0] TURLtransferURL="gsiftp://ds-815915.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea"
============================================================

It is important to remember the TURL which will be used in transfer command with the globus-url-copy command. After that, we can perform the file transfer:


So we can perform the transfer with a globus-url-copy command:

Code Block
languagebash
themeMidnight
-bash-4.2$ globus-url-copy gsiftp://ds-915.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea copia
-bash-4.2$ ls
ce_testp308.sub copia pass sleep.sh sleep.sub test.sub

Finally, to list the file in a directory you can use the command clientSRM Ls:

Code Block
languagebash
themeMidnight
-
Code Block
bash-4.2$ globus-url-copy /home/USER-SUPPORT/arendina/sleep.sub gsiftp://ds-815clientSRM Ls -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea

...

it:8444/virgo4

A user is also able to delete a file using the SRM clients. An easy example follows below:

Code Block
languagebash
themeMidnight
[arendina@ui-tier1 ~]$ clientSRM Rm
Code Block
-bash-4.2$ clientSRM Pd -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -t 51e58c63-afdd-4ccb-8a6d-8551b4261c33 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andreatest2402
============================================================
Sending PutDoneRm request to: httpg://storm-fe-archive.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
Request status:
  statusCode="SRM_SUCCESS"(0)
  explanation="All file requests are successfully completedfiles removed"
============================================================
SRM Response:
  arrayOfFileStatuses (size=1)
      [0] SURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andreatest2402"
      [0] status: statusCode="SRM_SUCCESS"(0)
                  explanation="SuccessFile removed"
============================================================

Similarly, in case of remote to local transfer the command is clientSRM PtG, where PtG stands for Prepare to Get, and to check the preparation status is  clientSRM SPtG:

More information on using SRM clients can be found here [18].

NB: For some VOs there is also the possibility to retrieve an https TURL using the ClientSRM utils. Indeed, this could be done adding the -T -P https option, for instance:

Code Block
languagebash
themeMidnight
[arendina@ui-tier1 ~]$ clientSRM PtP -T -P https -e httpg
Code Block
-bash-4.2$ clientSRM PTG -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archiveams.cr.cnaf.infn.it:8444/virgo4/prova_andrea
============================================================
Sending PtG request to: httpg:8444 -s srm://storm-fe-archiveams.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
/juno/storm/prova_andrea
========================================================================
Request status:
statusCode="SRM_REQUEST_QUEUED"(17)
Sending PtP request to: httpg://storm-fe-ams.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 17
Poll Flag 0
============================================================
SRM Response:
requestToken="fe633fd3-de07-4a3e-a388-3cc2adf1fd3a"
arrayOfFileStatuses (size=1)
[0]===
Request status:
  statusCode="SRM_REQUEST_QUEUED"(17)
explanation=""
[0] sourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
============================================================
SRM Response:
-bash-4.2$ clientSRM SPtG -e httpg  requestToken="de8af950-9869-4e50-891f-a8510c36cfdd"
  arrayOfFileStatuses (size=1)
      [0] SURL="srm://storm-fe-archiveams.cr.cnaf.infn.it:8444 -t fe633fd3-de07-4a3e-a388-3cc2adf1fd3a /juno/storm/prova_andrea"
      [0] status: statusCode="SRM_REQUEST_QUEUED"(17)
                  explanation=""
============================================================
Sending StatusPtG request to:

[arendina@ui-tier1 ~]$ clientSRM SPtP -e httpg://storm-fe-archiveams.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0 -s srm://storm-fe-ams.cr.cnaf.infn.it:8444/juno/storm/prova_andrea -t "de8af950-9869-4e50-891f-a8510c36cfdd"
============================================================
Request status:
statusCode="SRM_SUCCESS"(0)
explanation="All chunks successfully handled!"
Sending StatusPtP request to: httpg://storm-fe-ams.cr.cnaf.infn.it:8444
Before execute:
Afer execute:
Request Status Code 0
Poll Flag 0
============================================================
SRM Response:
arrayOfFileStatuses (size=1)
[0]Request status: statusCode="SRM_FILE_PINNED"(22)
explanation="srmPrepareToGet successfully handled!"
[0] sourceSURL="srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4/prova_andrea"
[0] fileSize=280
[0] transferURL="gsiftp://ds-915.cr.cnaf.infn.it:2811//storage/gpfs_virgo4/virgo4/prova_andrea
  statusCode="SRM_SUCCESS"(0)
  explanation="All chunks successfully handled!"
============================================================
SRM Response:
  arrayOfFileStatuses (size=

So we can perform the transfer with a globus-url-copy command:

Code Block
-bash-4.2$ globus-url-copy gsiftp://ds-9151)
      [0] SURL="srm://storm-fe-ams.cr.cnaf.infn.it:28118444/juno//storage/gpfs_virgo4/virgo4/prova_andrea copia
-bash-4.2$ ls
ce_testp308.sub copia pass sleep.sh sleep.sub test.sub

Finally, to list the file in a directory you can use the command clientSRM Ls:

Code Block
-bash-4.2$ clientSRM Ls -e httpg://storm-fe-archive.cr.cnaf.infn.it:8444 -s srm://storm-fe-archive.cr.cnaf.infn.it:8444/virgo4

...

storm/prova_andrea"
      [0] status: statusCode="SRM_SPACE_AVAILABLE"(24)
                  explanation="srmPrepareToPut successfully handled!"
      [0] TURL="https://ds-510.cr.cnaf.infn.it:8443/juno/storm/prova_andrea"
============================================================