Proxies

At INFN-Tier-1, valid WebDAV endpoints for the experiments’ storage areas are provided with StoRM WebDAV (third-party-copy supported).
The public page with the specific information to properly contact a CNAF endpoint is available at https://www.cnaf.infn.it/~usersupport/Webdav_SA.html.

Then, the most common WebDAV clients can be used to access the storage areas, namely browsers and command-line tools such as curl and gfal.

When StoRM WebDAV is used, VOMS proxies are supported only by command-line tool, and browsers can be used to navigate into the storage area content if anonymous read-only access is enabled (HTTP endpoint) or if VO users access by their X509 certificate is enabled (HTTPS endpoint).

A few useful commands follow and more info are available at [19].

With a valid voms-proxy:

[arendina@ui-tier1 ~]$ voms-proxy-init --voms juno
Contacting lcgvoms02.jinr.ru:15008 [/C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru] "juno"...
Remote VOMS server contacted succesfully.
Created proxy in /tmp/x509up_u10162.
Your proxy is valid until Fri Jul 02 05:42:21 CEST 2021

[arendina@ui-tier1 ~]$ voms-proxy-info --all
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it/CN=2090475310
issuer    : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
identity  : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
type      : RFC3820 compliant impersonation proxy
strength  : 2048
path      : /tmp/x509up_u10162
timeleft  : 11:59:35
key usage : Digital Signature, Key Encipherment
=== VO juno extension information ===
VO        : juno
subject   : /DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Andrea Rendina arendina@infn.it
issuer    : /C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms02.jinr.ru
attribute : /juno/Role=NULL/Capability=NULL
timeleft  : 11:59:35
uri       : lcgvoms02.jinr.ru:15008

  • Listing directories

    [arendina@ui-tier1 ~]$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno
rucio4juno
cronTest

  • Uploading a file

    [arendina@ui-tier1 ~]$ gfal-copy test.txt davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107
Copying file:///home/USER-SUPPORT/arendina/test.txt   [DONE]  after 0s

  • Downloading a file             

    [arendina@ui-tier1 ~]$ gfal-copy davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107 here
Copying davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107   [DONE]  after 0s

  • Removing a file

    [arendina@ui-tier1 ~]$ gfal-rm davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107
davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test0107  DELETED

Third-party-copies

In order to properly perform a third-party-copy between two endpoints which support the http protocol, one needs to set the BEARER_TOKEN up.
This token, also known as macaroon, is used to authenticate the user to the second endpoint. For this reason, the server to contact to get the token depends on whether the copy is in pull or push mode.

Actually, if both the endpoints are able to release a macaroon and the used gfal version is greater or equal to the 2.20 [28], the process is automatic.
Otherwise, if only one of the two endpoints can release a BEARER_TOKEN, or equivalently just one endpoint supports the http protocol, the user must request a macaroon to that endpoint.

Two easy examples follow below.

Pull-copy

Let's consider a TPC in pull mode between IHEP and CNAF: the user wants to pull a file from CNAF to IHEP.
The user has to authenticate itself with a valid voms-proxy to the first endpoint, in this case IHEP, and then IHEP will authenticate itself with a BEARER_TOKEN that the user had got previously from the second endpoint, namely the CNAF server (this happens because a pull-copy triggers a GET request). So, step by step:

  • The user requests a BEARER_TOKEN to the second endpoint, namely "xfer-archive" at CNAF.

    -bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

  • With this BEARER_TOKEN, the user can pull a file from CNAF to IHEP.

    -bash-4.2$ gfal-copy -v --copy-mode pull davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_Andrea davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big

Push-copy

Let's consider a TPC in push mode between IHEP and CNAF: the user wants to push a file from IHEP to CNAF.
Since the push-copy triggers a COPY request, the second endpoint to contact is CNAF again.

  • The user again requests a BEARER_TOKEN to the second endpoint, namely "xfer-archive" at CNAF.

    -bash-4.2$ export BEARER_TOKEN=$(curl -s --cacert ~/.globus/usercert.pem --cert $X509_USER_PROXY --key $X509_USER_PROXY --capath /etc/grid-security/certificates/ -X POST -d grant_type=client_credentials https://xfer-archive.cr.cnaf.infn.it:8443/oauth/token | jq -r .access_token)

  • Then, the user can push a file from IHEP to CNAF.

    -bash-4.2$ gfal-copy -v --copy-mode push davs://junoeos01.ihep.ac.cn:9000/eos/juno/user/rucio_test/test_Andrea_big davs://xfer-archive.cr.cnaf.infn.it:8443/juno/test_to_CNAF

Tokens

StoRM WebDAV also supports OpenID connect authentication and authorization on storage areas, so tokens can be used instead of proxies [20]. Dedicated IAM (Identity and Access Management) instances can be configured for the experiments upon requests (please contact User support).
The list of the storage areas grouped accoring to the token-issuers is available at the following link:
https://www.cnaf.infn.it/~usersupport/Webdav_token.html.

Also, a catch-all IAM instance is available at iam-t1-computing.cloud.cnaf.infn.it where specific groups can be created for different communities, and users can register and request group membership. In this case, exploiting the support for fine-grained authorization within StoRM WebDAV, users can browse their data after OIDC login, and/or they can access storage from command line using gfal (Gfal-utils) or curl with JWT tokens.

Once registered within IAM, an access token can be retrieved following the commands at 10 - Authentication and Authorization#10AuthenticationandAuthorization-JWT.

A useful full set of commands can be found here:

-bash-4.2$ eval `oidc-agent-service use`
Agent pid 17216
-bash-4.2$ oidc-add juno2
Enter decryption password for account config 'juno2':
success
-bash-4.2$ export BEARER_TOKEN=$(oidc-token juno2)
-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/juno/
rucio4juno
test_Andrea
cronTest

NB: gfal first tries to access the resource with the user X509 voms-proxy, which could lead to errors. 
In order to avoid this, the user must execute voms-proxy-destroy to properly switch to the JWT authentication.

Furthermore, an error like the following could appear, but it is just an irrelevant warning message:

(Davix::OpenSSL) Error: impossible to open /tmp/x509up_u10164:  : error:02001002:system library:fopen:No such file or directory

On the other hand, to switch back to the voms-proxy, one needs to unset the BEARER_TOKEN, namely:

-bash-4.2$ unset BEARER_TOKEN

-bash-4.2$ voms-proxy-init --voms belle
Contacting voms.cc.kek.jp:15020 [/C=JP/O=KEK/OU=CRC/CN=host/voms.cc.kek.jp] "belle"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_ucopy test https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/test_now

-bash-4.2$ gfal-ls davs://xfer-archive.cr.cnaf.infn.it:8443/belle
bellehttpd
TMP
CONTENT.stats
rucio4iddls

Curl examples

The curl command can also be used to perform data management operations with the StoRM WebDAV storage areas and tokens.
As shown above, some preliminary steps to retrieve a valid token are needed:

-bash-4.2$ eval `oidc-agent-service use`
25684
-bash-4.2$ oidc-add juno
Enter decryption password for account config 'juno':
success
-bash-4.2$ export BEARER_TOKEN=$(oidc-token juno)

The same operations performed using the gfal utils can be reproduced with the curl command. A few useful examples follow below:

  • Listing directory

    -bash-4.2$ curl -H "Authorization: Bearer $BEARER_TOKEN" --capath /etc/grid-security/certificates/ https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/

  • Uploading a file

    -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $BEARER_TOKEN" -X PUT https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_Andrea --data-binary "@test"

    where test is the file in the user local folder.

  • Downloading a file      

    -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $BEARER_TOKEN" https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_Andrea -o local_copy
       % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                        Dload  Upload   Total   Spent    Left  Speed
       100    10  100    10    0     0     22      0 --:--:-- --:--:-- --:--:--    22 (edited)

  • Creating a directory

    -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Authorization: Bearer $BEARER_TOKEN" -X MKCOL https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_dir

  • Removing a file or a directory

    -bash-4.2$ curl --capath /etc/grid-security/certificates -H "Content-Type: text/csv" -H "Authorization: Bearer $TOKEN" -X DELETE https://xfer-archive.cr.cnaf.infn.it:8443/juno-test/test_dir
  • No labels