Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Preliminary information

  • The backup scripts are stored in /u3/scripts;

  • The log files for these scripts are stored in /u3/scripts/logs; These logs are rotated on dafnenfs from logrotate (check last point);

  • The script u2_backup.sh is responsible to create a backup of /u2 in /u3/u2_backup and launch nfs_rsync.sh script to rsync data to nascoda1. The u2_backup.sh script is launched by crontab on host dafnenfs every day (see Table 2 for details). The script sends an e-mail to andrea.michelotti@lnf.infn.it francesco.galletti@lnf.infn.it to notify the exit status.
  • The script nfs_rsync.sh is responsible to rsync the data from the host where it is launched to nascoda1. It sends an e-mail to andrea.michelotti@lnf.infn.it francesco.galletti@lnf.infn.it to notify the backup exit status. It is launched by crontab on host danteweb and vldafneafp every day (see Table 2 for details);
  • The host dafnenfs share /u3/scripts; danteweb and vldafneafp mount that folder via NFS on boot (folder present in /etc/fstab);
  • The host dafnenfs has a custom configuration for logrotate present in /etc/logrotate.d/nfs_rsync to rotate log files generated by these scripts. It will rotate all log files every day, keeping them compressed in /u3/scripts/logs/old folder for 5 days.

Git repository for backup scripts

Link / Path

Note

link: https://baltig.infn.it/lnf-da-control/dcs-backup-scripts

branch : main

Script in repository:

  • nfs_rsync.sh
  • u2_backup.sh

Data

Control system data was saved in a Sun Storagetek 2550 Disk Array (LAT room) connected with two fiber optic paths to beatrix and virgil, the old production cluster. Now it's all stored and exported to us from the IT department of LNF; These are the folders in which data is organized: 

HostFolder

(warning) Page under construction (warning)

Dati

I dati del sistema di controllo sono salvati all'interno del Disk Array Sun Storagetek 2550 installato in sala LAT e connesso con doppio path in fibra ottica verso ognuno dei due nodi del cluster di produzione, virgil e beatrix. I folder condivisi dal cluster e da salvare su backup sono i seguenti:

Logical Volume (Disk Array)Mount Folder (Nodo Attivo del Cluster)
Backup Folder (nascoda1/2)
DescrizioneVGShared-LVDiskless
Description
dafnenfs/etcserver_data/dafnenfs/etcConfiguration files
dafnenfs/data/disklessdisklessFilesystem
per le
for VMIC diskless clients
VGShared-LVForceUsr
dafnenfs/data/export/exec/Solaris_9_sparc.all/usrforce_usrUser Filesystem
per le
for Force diskless clients
VGShared-LVForceRoot
dafnenfs/data/export/rootforce_rootRoot Filesystem
per le
for Force diskless clients
VGShared-LVForceSwap
dafnenfs/data/export/swapforce_swapSWAP
per le
for Force diskless clients
VGShared-LVU1
dafnenfs/data/u1u1
VGShared-LVU2

dafnenfs/data/u2u2
Sistema di controllo
Control system, home folders (
per
dafne, vmic e VMs)
ed altri fileVGShared-LVU3
and other files
dafnenfs/data/u3u3LVTemp,
backup di /u2 ed altri file sensibiliVGShared-LVMySQL/var/lib/mysqlmysqlDatabase files su cluster (non usato)VGShared-LVApache/var/www/htmlapacheWeb files su cluster (non usato)VGShared-LVSiddharta/siddharta/dataFromSiddhartasiddharta/dataFromSiddhartaDati prodotti da Siddharta necessari a DafneVGShared-LVDafne/siddharta/dataFromDafnesiddharta/dataFromDafneDati prodotti da Dafne necessari a Siddharta

Configurazioni

Le configurazioni da salvare sono le seguenti:

/data/u2 backup and other sensible files
dafnenfs/siddhartasiddhartaData written from Siddharta experiment and from DAFNE
dafnenfs/data/slmonitorslmonitorFolder for A. Stella
ServerFolderBackup Folder (nascoda1/2)Descrizione

danteweb/etcserver_data/danteweb/etc
File di configurazione di Apache, MySQL, Memcached e tutto ciò che può servire a fare un rollback
Configuration files
danteweb/optserver_data/danteweb/opt
Script di servizio e dati utili, compresa una snapshoot del crontab installato
Service scripts and data, including a crontab snapshot
danteweb/var/lib/
www
mysqlserver_data/danteweb/
var/wwwFile dei servizi web offerti da danteweb (devilmanager, WCMDATA, devilinfo, etc)
mysqlDatabase files
danteweb/var/
lib/mysql
wwwserver_data/danteweb/var/
lib/mysqlFile dei databases utilizzati dal controllo e dai servizi webbeatrix
wwwWeb files

vldafneafp/etcserver_data/
beatrix
vldafneafp/etc
File di configurazione dei servizi e del clusterbeatrix
Configuration files
vldafneafp/srv
/opt/scripts
server_data/
beatrix
vldafneafp/
opt/scriptsScripts di servizio e crontab del servervirgil/etcserver_data/virgil/etcFile di configurazione dei servizi e del clustervirgil/opt/scriptsserver_data/virgil/opt/scriptsScripts di servizio e crontab del server

Date ed orari di backup

I crontab di backup sono i seguenti:

...

srvAPPLE DEVIL's FILESYSTEM data shared via AFP

Table 1 - data folders

Backup planning

Backup crontab are planned as Table 2 reports:

ServerBackup frequencyTimeDescription
dafnenfsdaily01:30Create a backup of u2 in u3/u2_backup and then launch rsync script
dantewebdaily01:10Launch rsync script
vldafneafpdaily02:01Launch rsync script

Table 2 - Backup planning

Snapshot and replica

Backups are planned on

*eseguito solo dal nodo attivo del cluster

Snapshot e repliche

I backup sono effettuati sul NAS nascoda1.lnf.infn.it (Sala LAT ) e replicati sul NAS nascoda2room) and rsynced on NAS nasda.lnf.infn.it (Laboratorio Controlli, Ed. Control Lab, Building 2, Stanza Room 117) con il seguente schema (comprensivo di snapshot dei singoli volumi)with following planning:

Source NAS
Sorgente
Nas Destinazione
Destination NAS
Operazione
Operation
Frequenza
Frequency
Orario
Time
Descrizione
Description
nascoda1
locale
local
Snapshot del volume dei datiGiornaliera00:05Snapshot mantenute per 1 mesenascoda1nascoda2Snapshot e Replica del volume di sistema del NASGiornaliera00:55Snapshot del volume di sistema del NAS locale e replica su nascoda2nascoda1localeSnapshot del volume delle configurazioniSettimanale (Domenica)01:05Snapshot mantenute per 1 mesenascoda1nascoda2Sync del volume dei Dati e delle ConfigurazioniGiornaliera04:00rsync da nascoda1 a nascoda2 di tutti i volumi (dati e configurazioni)nascoda2localeSnapshot del volume dei datiGiornaliera00:05Snapshot mantenute per 1 mesenascoda2nascoda1Snapshot e Replica del volume di sistema del NASGiornaliera00:55Snapshot del volume di sistema del NAS locale e replica su nascoda1nascoda2localeSnapshot del volume delle configurazioniSettimanale (Domenica)01:05Snapshot mantenute per 1 mese

Backup manuale mensile

Data volume snapshotSmart Versioning-Hourly snapshots: 24
Daily snapshots: 7
Weekly snapshots: 5
Monthly snapshots: 12
nascoda1nasdaData volume syncDaily04:00rsync data from nascoda1 to nasda/nascoda folder

Table 3 - Snapshot and replica planning

Recover data from backup

Read this documentation page.

Backup on disaster recovery USB disk

Disaster recovery diskImage AddedBackup notesImage Added

To backup DCS from nasda to the disaster recovery disk, this is the procedure to be executed:

  1. Connect the disk to the front usb port of nasda and wait for the blue usb led to become steady on;
  2. Keep pressed the "Copy" button in front of nasda for about 3 seconds, then release the button and check that the blue usb led begins to blink and a single beep coming from nasda. When the backup is finished, the blue usb led will be steady on;
  3. To safely disconnect the disk, keep pressed the "Copy" button for about 6 seconds, until the blue usb led begins to blink and two beeps come from nasda; then release the button and wait for the blue usb led to turn off. Then, disconnect the disk;
  4. Write the date and the name of who made the backup on the "backup notes" present inside the disk box and keep the disk in a safe place;

There are two videos showing how to start the backup and how to disconnect the disk:

View file
name01-inserimento-disco-e-creazione-backup.mp4
height250
View file
name02-rimozione-disco.mp4
height250

Excerpt Include
Recover files from backup
Recover files from backup
Ogni mese si effettua un backup manuale (in formato .tar.gz) di tutti i folder dei dati e si salva nella cartella condivisa "DCS Backups" su nascoda1.lnf.infn.it