Data stored on tape can also be recalled with HTTP protocol, thanks to the development of a common HTTP REST interface within the WLCG community in support of the transition towards srm-less recalls. 

In particular, in the context of the StoRM project developed at INFN-CNAF, a new component of such service, named StoRM Tape REST API, is used, and it has been integrated within the existing infrastructure, in particular with the Grid-Enabled Mass Storage System (GEMSS). Such API is RESTful and consists of a single endpoint handling different operations: a bulk request to stage many tape-stored files, making them available with disk latency; tracking the progress of a previous stage request; cancellation and deletion of previous requests; access to staged file metadata.

Paths to the data stored on tape will be provided by CNAF User Support team (user.support@list.cnaf.infn.it).

The StoRM Tape REST API supports authentication mechanisms based on VOMS proxies and OAuth2 tokens (see Data transfers using http endpoints to learn how to get a valid token).
In this regard, to properly switch from the first one to the latter, or vice versa, it is enough to replace

--cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID

with

-H "Authorization: Bearer $BEARER_TOKEN"

in all the curl commands below, where BEARER_TOKEN is the exported variable containing the JWT token.

Check if a file is on disk/tape (archiveinfo)

To know the locality (tape, disk, both) of a given file (or a set of files), a JSON file like this one is needed:

{
  "paths": ["/tape/file"]
}

Then, it is enough to issue:

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/archiveinfo -d @paths.json

Recall files from tape (stage request)

A user can issue a recall of a file (or of a set of files) launching:

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage -d @stage_request.json -i

where stage_request is a JSON file of the following shape

{
  "files": [
    {
      "path": "/tape/file"
    }
  ]
}

The output of the stage request will show a location/requestID which can be used to check the status of the request (STARTED/SUBMITTED/COMPLETED).
For instance:

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage -d @stage_request.json -i
HTTP/1.1 201 Created
Server: nginx/1.24.0
Date: Thu, 13 Sep 2023 13:08:19 GMT
Content-Type: application/json
Content-Length: 52
Connection: keep-alive
Location: https://tape-atlas.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2
{"requestId":"56ec5421-5921-4536-aae6-38003aad03c2"}

and then

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2
{"id":"56ec5421-5921-4536-aae6-38003aad03c2","createdAt":1689253699,"startedAt":0,"completedAt":0,"files":[{"path":"/tape/file","state":"SUBMITTED"}]}

After being recalled, files get "pinned" to disk for 3 days, with such pin time being configurable by our StoRM admins.

Delete a stage request

To delete a stage request whose ID is known, a user can execute

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID -X DELETE https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2

In case he/she would rather delete only those recalls relative to a subset of the files of the original stage request, such files must be stated in a JSON file.
For example:

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2/cancel -d @paths.json

where paths.json is

{
  "paths": ["/tape/file"]
}

Release a file (not yet supported)

If a file is both on tape and disk, a user can ask to remove the copy on disk buffer by issuing:

curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$ID --cert /tmp/x509up_u$ID --key /tmp/x509up_u$ID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/release/56ec5421-5921-4536-aae6-38003aad03c2/ -d @paths.json
  • No labels