Page History

Anchor
Proxy WebDAV Proxy WebDAV

Proxies

At INFN-Tier-1, valid WebDAV endpoints for the experiments’ storage areas are provided with StoRM WebDAV (third-party-copy supported) or Apache.
The public page with all the features to properly contact a CNAF endpoint is available at https://www.cnaf.infn.it/~usersupport/Webdav_SA.html.

...

With a valid voms-proxy:

[arendina@ui-tier1 ~]$ voms-proxy-init --voms juno
Contacting lcgvoms02.jinr.ru:15008 [/C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru] "juno"...
Remote VOMS server contacted succesfully.
Created proxy in /tmp/x509up_u10162.
Your proxy is valid until Fri Jul 02 05:42:21 CEST 2021

[arendina@ui-tier1 ~]$ voms-proxy-info --all
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=2090475310
issuer    : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity  : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type      : RFC3820 compliant impersonation proxy
strength  : 2048
path      : /tmp/x509up_u10162
timeleft  : 11:59:35
key usage : Digital Signature, Key Encipherment
=== VO juno extension information ===
VO        : juno
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer    : /C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru
attribute : /juno/Role=NULL/Capability=NULL
timeleft  : 11:59:35
uri       : lcgvoms02.jinr.ru:15008

...

• Listing directory
  

  Code Block
  language bash theme Midnight
  [arendina@ui-tier1 ~]$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno rucio4juno cronTest

  
  

• Upload
  

  Code Block
  language bash theme Midnight
  [arendina@ui-tier1 ~]$ gfal-copy test.txt davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 Copying file:///home/USER-SUPPORT/arendina/test.txt [DONE] after 0s

  
  

• Download             
  

  Code Block
  language bash theme Midnight
  [arendina@ui-tier1 ~]$ gfal-copy davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 here Copying davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 [DONE] after 0s

  
  

• Removing a file
  

  Code Block
  language bash theme Midnight
  [arendina@ui-tier1 ~]$ gfal-rm davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 DELETED

  
  

...

• I ask a BEARER_TOKEN to the second endpoint, namely "xfer-archive" at CNAF.

  Code Block
  language bash theme Midnight
  -bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

  
  

• I can pull a file from CNAF to IHEP.

  Code Block
  language bash theme Midnight
  -bash-4.2$ gfal-copy -v --copy-mode pull davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_Andrea davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big

  
  

...

• I ask a BEARER_TOKEN to the second endpoint, as above.

  Code Block
  language bash theme Midnight
  -bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

  
  

• Then, I can push a file from IHEP to CNAF.

  -

  Code Block
  language	bash theme Midnight
  -bash-4.2$ gfal-copy -v --copy-mode push davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_to_CNAF

  
  

Anchor
Tokens WebDAV Tokens WebDAV

Tokens

StoRM WebDAV also supports OpenID connect authentication and authorization on storage areas, so tokens can be used instead of proxies [20]. Dedicated IAM (Identity and Access Management) instances can be configured for the experiments upon requests (please contact the user support).
The list of the storage areas divided into the proper token-issuers is available at the following link:
https://www.cnaf.infn.it/~usersupport/Webdav_token.html.

...

At first, we have to start the process with the command

[arendina@ui-tier1 ~]$ eval `oidc-agent-service use`
Agent pid 20578

Then, one needs to register his own client (this has to be done just the first time to create a new local IAM client):

[arendina@ui-tier1 ~]$ oidc-gen -w device

...

Whereas, from the second time it is enough typing:

[arendina@ui-tier1 ~]$ oidc-add <your-client-name>
Enter decryption password for account config '<your-client-name>':
success

To get the access token and save it an environment variable:

[arendina@ui-tier1 ~]$ TOKEN=$(oidc-token <your-client-name>)

The token will be valid for 60 minutes. It can be obtained a new token issuing the oidc-token command again.
At the end, stop oidc-agent daemon:

oidc-agent-service stop

Moreover, to properly use the gfal tools it occurs to set the BEARER_TOKEN environment variable up.

A useful full set of commands can be found here:

-bash-4.2$ eval `oidc-agent-service use`
Agent pid 17216
-bash-4.2$ oidc-add juno2
Enter decryption password for account config 'juno2':
success
-bash-4.2$ export BEARER_TOKEN=$(oidc-token juno2)
-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno/
rucio4juno
test_Andrea
cronTest

...

Furthermore, an error like this could appear, but it is just an irrelevant warning message:

(Davix::OpenSSL) Error: impossible to open /tmp/x509up_u10164:  : error:02001002:system library:fopen:No such file or directory

On the other hand, to switch back to the voms-proxy it occurs to unset the BEARER_TOKEN, namely:

-bash-4.2$ unset BEARER_TOKEN

-bash-4.2$ voms-proxy-init --voms belle
Contacting voms.cc.kek.jp:15020 [/C=JP/O=KEK/OU=CRC/CN=host/voms.cc.kek.jp] "belle"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_ucopy test https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/test_now

-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/belle
bellehttpd
TMP
CONTENT.stats
rucio4iddls

...

A user can also use the curl command to make data management with the StoRM WebDAV storage areas and tokens.
As shown before, some steps to retrieve a valid token are needed:

-bash-4.2$ eval `oidc-agent-service use`
25684
-bash-4.2$ oidc-add juno
Enter decryption password for account config 'juno':
success
-bash-4.2$ export BEARER_TOKEN=$(oidc-token juno)

...

• Listing directory
  

  Code Block
  language bash theme Midnight
  -bash-4.2$ curl -H "Authorization: Bearer $BEARER_TOKEN" --capath /etc/grid-security/certificates/ https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/

  
  

• Upload
  

  Code Block
  language bash theme Midnight
  -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $BEARER_TOKEN" -X PUT https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_Andrea --data-binary "@test"

  where test is the file in the user local folder.
  
  

• Download             
  

  Code Block
  language bash theme Midnight
  -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $BEARER_TOKEN" https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_Andrea -o local_copy % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10 100 10 0 0 22 0 --:--:-- --:--:-- --:--:-- 22 (edited)

  
  

• Create a directory
  

  Code Block
  language bash theme Midnight
  -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Authorization: Bearer $BEARER_TOKEN" -X MKCOL https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_dir

  
  

• Removing a file or a directory
  

  Code Block
  language bash theme Midnight
  -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $TOKEN" -X DELETE https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_dir

...