In order to work with data stored on tapes, the files must first be present on a disk buffer. Allocation of disk space on such buffer is automatically handled via retention and migration policies. There are two separate middleware that allow to handle files on tape:
- StoRM WebDAV allows to write and read files to and from the disk buffer.
- StoRM Tape REST API allows to request the staging of files from tape to the disk buffer.
The StoRM Tape REST API consists of a single endpoint handling different operations: bulk stage request to stage many tape-stored files, making them available with disk latency; tracking of the progress of a previous stage request; cancellation and deletion of previous requests; access to the metadata of the staged files.
The default pin time, that is the time a file remains "pinned" before being removed from disk buffer, is of 3 days.
Accessing the Tape REST API using gfal2
Be sure to have the proper authentication key, either a VOMS proxy or a JWT, depending on the specific configuration of the storage area (10 - Authentication and Authorization).
Before reading a file on tape, it must be present on the disk buffer.
For assessing the file availability, a user can issue the following command:
[ashtimmerman@ui-tier1 ~]$ gfal-xattr davs://xfer-archive.cr.cnaf.infn.it:8443/<file_path> user.status
which returns the string ONLINE_AND_NEARLINE if the file is on both tape and disk buffer, NEARLINE if the file is on tape only, or ONLINE if the file is on disk buffer only.
To bring online a file that is only on tape, i.e. in NEARLINE state:
[ashtimmerman@ui-tier1 ~]$ gfal-bringonline --polling-timeout 6000 https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> Bringonline token: b30e91d1-e7f7-417e-878b-7588deadbd92 https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> QUEUED Request queued, sleep 1 seconds... https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> QUEUED Request queued, sleep 2 seconds... https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> QUEUED Request queued, sleep 4 seconds... https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> READY
The --polling-timeout flag activates the polling mode of gfal-bringonline command which continuously checks the status of the file and exits when the file is ready to be transferred. After that, the file is in the ONLINE_AND_NEARLINE state:
[ashtimmerman@ui-tier1 ~]$ gfal-xattr https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> user.status ONLINE_AND_NEARLINE
Eventually, the file can be read using gfal-copy:
[ashtimmerman@ui-tier1 ~]$ gfal-copy https://xfer-archive.cr.cnaf.infn.it:8443/<file_path> read-test
In general, it is highly recommended to recall files in bulk. For this purpose, the --from-file option of gfal-bringonline allows to enqueue the stages from a list:
[ashtimmerman@ui-tier1 ~]$ gfal-bringonline --from-file ./list
Using cURL
The StoRM Tape REST API supports authentication mechanisms based on VOMS proxies and OAuth2 tokens (see 10 - Authentication and Authorization to learn how to get a valid token or proxy).
In this regard, to properly switch from the first one to the latter, or vice versa, it is enough to replace:
--cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID
with:
-H "Authorization: Bearer $BEARER_TOKEN"
in all the curl commands below, where BEARER_TOKEN is the exported variable containing the JWT.
Check if a file is on disk/tape (archiveinfo)
To know the locality (tape, disk, both) of a given file (or a set of files), a JSON file like this one is needed:
{
"files": [
{
"path": "/tape/file"
}
]
}
Then, it is enough to issue:
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/archiveinfo -d @paths.json
Recall files from tape (stage request)
A user can issue a recall of a file (or of a set of files) launching:
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage -d @stage_request.json -i
where stage_request is a JSON file of the following shape
{
"files": [
{
"path": "/tape/file"
}
]
}
The output of the stage request will show a location/requestID which can be used to check the status of the request (STARTED/SUBMITTED/COMPLETED).
For instance:
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage -d @stage_request.json -i
HTTP/1.1 201 Created
Server: nginx/1.24.0
Date: Thu, 13 Sep 2023 13:08:19 GMT
Content-Type: application/json
Content-Length: 52
Connection: keep-alive
Location: https://tape-atlas.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2
{"requestId":"56ec5421-5921-4536-aae6-38003aad03c2"}
and then
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2
{"id":"56ec5421-5921-4536-aae6-38003aad03c2","createdAt":1689253699,"startedAt":0,"completedAt":0,"files":[{"path":"/tape/file","state":"SUBMITTED"}]}
After being recalled, files get "pinned" to disk for 3 days, with such pin time being configurable by our StoRM admins.
Delete a stage request
To delete a stage request whose ID is known, a user can execute
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID -X DELETE https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2
In case he/she would rather delete only those recalls relative to a subset of the files of the original stage request, such files must be stated in a JSON file.
For example:
curl --capath /etc/grid-security/certificates --cacert /tmp/x509up_u$UID --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://tape-<endpoint>.cr.cnaf.infn.it:8443/api/v1/stage/56ec5421-5921-4536-aae6-38003aad03c2/cancel -d @paths.json
where paths.json is
{
"files": [
{
"path": "/tape/file"
}
]
}