Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Preliminary information

  • The backup scripts are stored in /u3/scripts;

  • The log files for these scripts are stored in /u3/scripts/logs; These logs are rotated on dafnenfs from logrotate (check last point);

  • The script u2_backup.sh is responsible to create a backup of /u2 in /u3/u2_backup and launch nfs_rsync.sh script to rsync data to nascoda1. The u2_backup.sh script is launched by crontab on host dafnenfs every day (see Table 2 for details). The script sends an e-mail to andrea.michelotti@lnf.infn.it francesco.galletti@lnf.infn.it to notify the exit status.
  • The script nfs_rsync.sh is responsible to rsync the data from the host where it is launched to nascoda1. It sends an e-mail to andrea.michelotti@lnf.infn.it francesco.galletti@lnf.infn.it to notify the backup exit status. It is launched by crontab on host danteweb and vldafneafp every day (see Table 2 for details);
  • The host dafnenfs share /u3/scripts; danteweb and vldafneafp mount that folder via NFS on boot (folder present in /etc/fstab);
  • The host dafnenfs has a custom configuration for logrotate present in /etc/logrotate.d/nfs_rsync to rotate log files generated by these scripts. It will rotate all log files every day, keeping them compressed in /u3/scripts/logs/old folder for 5 days.

Git repository for backup scripts

Link / Path

Note

link: https://baltig.infn.it/lnf-da-control/dcs-backup-scripts

branch : main

Script in repository:

  • nfs_rsync.sh
  • u2_backup.sh

...

Data

Control system data was saved in a Sun Storagetek 2550 Disk Array (LAT room) connected with two fiber optic paths to beatrix paths to beatrix and virgil, the old production cluster. Now it's all stored and exported to us from the IT department of LNF; These are the folders in which data is organized: 

/varlib/mysql
HostFolderBackup Folder (nascoda1/2)Description
ServerFolderBackup Folder (nascoda1/2)Description
dafnenfs/etcserver_data/dafnenfs/etcConfiguration files
dafnenfs/data/disklessdisklessFilesystem for VMIC diskless clients
dafnenfs/data/export/exec/Solaris_9_sparc.all/usrforce_usrUser Filesystem for Force diskless clients
dafnenfs/data/export/rootforce_rootRoot Filesystem for Force diskless clients
dafnenfs/data/export/swapforce_swapSWAP for Force diskless clients
dafnenfs/data/u1u1
dafnenfs/data/u2u2Control system, home folders (dafne, vmic e VMs) and other files
dafnenfs/data/u3u3LVTemp, /data/u2 backup and other sensible files
dafnenfs/siddharta/dataFromSiddhartasiddharta/dataFromSiddhartaData written from Siddharta experiment useful for DAFNEdafnenfs/siddharta/dataFromDafnesiddharta/dataFromDafneData written from DAFNE useful for Siddharta experimentand from DAFNE
dafnenfs/data/backup_linacslmonitor-slmonitorFolder for A. Stellavllinacctr001 VM backup

danteweb/mysqlDatabase files (backup disabled; to be re-enabled soon)
danteweb/var/wwwapacheWeb files (backup disabled; to be re-enabled soon)

Table 1 - data folders

Configurations

Configuration files to be saved:

/Apache, MySQL, Memcached config files and all needed for a rollback snapshootdafnenfsetcnagiosvlmonitor01etcnagiosNagios configuration
etcserver_data/danteweb/etcConfiguration files
danteweb/optserver_data/danteweb/optService scripts and data, including a crontab snapshot
danteweb/etcvar/lib/mysqlserver_data/dafnenfsdanteweb/etcmysqlConfiguration Database files (never stored)
vlmonitor01danteweb/var/wwwserver_data/danteweb/var/wwwWeb files

vlmonitor01vldafneafp/var/spool/nagiosetcserver_data/vlmonitor01/var/spool/nagiosNagios home (with public/private keys)vldafneafp/etcConfiguration files
vldafneafp/srvvlmonitor01/opt/nagios_custom_pluginsserver_data/vlmonitor01/opt/nagios_custom_pluginsNagios custom pluginsvldafneafp/srvAPPLE DEVIL's FILESYSTEM data shared via AFP

Table 1 - data Table 2 - configuration folders

Backup planning

Backup crontab are planned as Table 3 2 reports.:

ServerBackup frequencyTimeDescription
dantewebTo be implemented(warning)dafnenfsTo be implemented(warning)vlmonitor01To be implemented(warning)
dafnenfsdaily01:30Create a backup of u2 in u3/u2_backup and then launch rsync script
dantewebdaily01:10Launch rsync script
vldafneafpdaily02:01Launch rsync script

Table 2 Table 3 - Backup planning

Snapshot and replica

Backups have to be Backups are planned on NAS nascoda1.lnf.infn.it (LAT room) and rsynced on NAS nascoda2 nasda.lnf.infn.it (Control Lab, Building 2, Room 117) with following planning (including single volume snapshots):

Source NASDestination NASOperationFrequencyTimeDescription
nascoda1localData volume snapshotDaily00:05Snapshot kept fpr 1 month
nascoda1nascoda2NAS system volume snapshot and replicaDaily00:55Local NAS system volume snapshot and replica on nascoda2
nascoda1localConfiguration volume snapshotWeekly (Sunday)01:05Snapshot kept fpr 1 month
Smart Versioning-Hourly snapshots: 24
Daily snapshots: 7
Weekly snapshots: 5
Monthly snapshots: 12
nascoda1nasdaData nascoda1nascoda2Data and configuration volume syncDaily04:00rsync data from nascoda1 to nascoda2 of all volumes
nascoda2localData volume snapshotDaily00:05Snapshot kept fpr 1 month
nascoda2nascoda1NAS system volume snapshot and replicaDaily00:55Local NAS system volume snapshot and replica on nascoda1
nascoda2localConfiguration volume snapshotWeekly (Sunday)01:05Snapshot kept fpr 1 month

Table 4 - Snapshot and replica planning

Backup manuale mensile

nasda/nascoda folder

Table 3 - Snapshot and replica planning

Recover data from backup

Read this documentation page.

Backup on disaster recovery USB disk

Disaster recovery diskImage AddedBackup notesImage Added

To backup DCS from nasda to the disaster recovery disk, this is the procedure to be executed:

  1. Connect the disk to the front usb port of nasda and wait for the blue usb led to become steady on;
  2. Keep pressed the "Copy" button in front of nasda for about 3 seconds, then release the button and check that the blue usb led begins to blink and a single beep coming from nasda. When the backup is finished, the blue usb led will be steady on;
  3. To safely disconnect the disk, keep pressed the "Copy" button for about 6 seconds, until the blue usb led begins to blink and two beeps come from nasda; then release the button and wait for the blue usb led to turn off. Then, disconnect the disk;
  4. Write the date and the name of who made the backup on the "backup notes" present inside the disk box and keep the disk in a safe place;

There are two videos showing how to start the backup and how to disconnect the disk:

View file
name01-inserimento-disco-e-creazione-backup.mp4
height250
View file
name02-rimozione-disco.mp4
height250

Excerpt Include
Recover files from backup
Recover files from backup
Ogni mese si effettua un backup manuale (in formato .tar.gz) di tutti i folder dei dati e si salva nella cartella condivisa "DCS Backups" su nascoda1.lnf.infn.it