Remote Backup

This section describes the procedure for using DCAP Central Remote Backup.

Terminology

DCAP Central-machine - The machine where the DCAP Central system is running, and should be backed up (a.k.a. The source system).

Backup-machine - The system where the sonar remote backup is installed.

Prerequisites

Ensure that SELinux is set to Permissive:

setenforce permissive

Installation

Installation using a tarball

  1. Download the tarball to the Backup-machine.

  2. Un-tar the tarball to a local folder i.e.:

    tar -xvf rhel7.x_sonarbackup_installer_*.tar.gz
    
  3. Copy the setup script to the local folder:

    cp sonarbackup_rhel7_local_repo/*.sh ./
    
  4. Run the installer:

    sudo ./sonarbackup_rhel7_install.sh | & tee sonarbackup-install.out
    
  5. Check the sonarbackup-install.out for errors:

    grep -inE "error|fail|not|non|missing|warning" sonarbackup-install.out
    

Installation using package repository

  1. Enable Extra Packages for Enterprise Linux (epel):

    sudo rpm -U https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    
  2. Add the jsonar RHEL7 repository:

    sudo cat <<'EOF' | sudo tee /etc/yum.repos.d/jsonar.repo
    [Jsonar]
    name=Jsonar Repository
    baseurl=http://<your user name>:<your password>@rpm.jsonar.com/
    enabled=1
    priority=1
    gpgcheck=0
    EOF
    
  3. Install sonarremotebackup:

    sudo yum -y install sonarremotebackup
    

Setting up the backup system

  1. Copy cert.pem file (located in: /etc/sonar/ssl/client/admin/) from DCAP Central-machine to the Backup-machine (to: /etc/sonar/backup folder).

  2. Chown the cert.pem on the Backup-machine:

    sudo chown sonarbackup:sonar /etc/sonar/backup/cert.pem
    
  3. Add the rsa key from the Backup-machine to the sonarw authorized_keys:

    3.1. Check if the .ssh directory exists in the SONAR_HOME directory on the DCAP Central-machine,

    (default location is: /var/lib/sonarw):

    sudo ls -la /var/lib/sonarw/

    3.2. If it doesn’t exist, add the “.ssh” folder:

    sudo mkdir /var/lib/sonarw/.ssh

    3.3. If the “authorized_keys” file doesn’t exist, add it:

    sudo touch /var/lib/sonarw/.ssh/authorized_keys

    3.4. Chown the newly created folder and file:

    sudo chown -R sonarw:sonar /var/lib/sonarw/.ssh

    3.5. Copy the content of “/etc/sonar/backup/backup-key.pub” (from the Backup-machine), into the “authorized_keys” file in the sonarw home “.ssh” folder

    3.6. Test ssh from the Backup-machine to the DCAP Central-machine:

    sudo -u sonarbackup bash

    ssh -i /etc/sonar/backup/backup-key sonarw@<DCAP Central-ip-address>

    • Don’t be alarmed if it says “Failed to add the host to the list of known hosts”.

    3.7. Exit twice(from the ssh and the sonarbackup user shell):

    exit exit

  4. Edit the config file on the Backup-machine:

    sudo vi /etc/sonar/backup.conf
    

and set the next variables:

sonar_client_uri = mongodb://CN%%3Dadmin@<DCAP Central-machine-ip-address>:27117/admin?authSource=%%24external&authMechanism=PLAIN&certfile=%%2Fetc%%2Fsonar%%2Fbackup%%2Fcert.pem

remote_server_uri = sonarw@<DCAP Central-machine-ip-address>

admin_email = <system admin email>

backup_dir = /path/to/backup/folder

reports_dir = /path/to/reports/folder/on/DCAP Central-machine

  • By default it will backup all the databases and collections on the DCAP Central-machine, in case of backing up specific databases or collections, refer to the “backup.conf” file for instructions.
  1. On the DCAP Central-machine, edit “sonard.conf” file:

    sudo vi /etc/sonar/sonard.conf
    

and set the next variables:

bind_ip = 0.0.0.0
accept_connections_from = 127.0.0.1/32,<Backup-machine-ip-address>/32
  1. Restart sonard service (on the DCAP Central-machine):

    sudo systemctl restart sonard
    
  2. On the Backup-machine, start the remote backup service timer (will run backup daily by default):

    sudo systemctl start sonarremotebackup.timer
    

To change backup timing:

A. Create a copy of the timer in “/etc/systemd/system”:

    cp /usr/lib/systemd/system/sonarremotebackup.timer /etc/systemd/system

B. Edit the timer and set to the desired schedule:

    vi /etc/systemd/system/sonarremotebackup.timer

C. Change the [Timer] section to the desired schedule examples:

Every 12 hours:

    OnBootSec=15min
    OnActiveSec=12h

At a specific hour:

    OnCalendar=*-*-* 12:00:00

Every 12 Hr at specific times (1 AM & 1 PM):

    OnCalendar=*-*-* 01/13:00:00

* **For additional options refer to rhel7 systemd timers documentation**

D. Run:

    sudo systemctl daemon-reload
    sudo systemctl restart sonarremotebackup.timer
  1. To force one run of the backup, do:

    service sonarremotebackup start
    
  • The backup log file can be found at: /var/log/sonar/sonar-backup.log

Setting up a local backup

It is possible to set up a backup on the same machine that is running DCAP Central. Follow the above steps treating the single machine as both the DCAP Central machine and the backup machine, and use the localhost ip (e.g. 127.0.0.1) as the DCAP Central-ip-address. While having the backup on the same machine is not recommended in the general case (because on a total machine failure, both the backup and the main database will be lost), this functionality can provide easier integration with existing backup strategies. In particular the backup directory can (and should) be configured to be a mount point to a location that the user can replicate, store on tape for long term etc…

Upgrade from sonarBackup package to the newer sonarremotebackup

1.Upgrade DCAP Central

2.Setup the new sonarremotebackup (use different backup folder then the old backup).

3.Run sonarremotebackup and check that it finished without any errors.

4.Delete the old backup directory.

Full Machine Restore from Backup

This section explains how to restore a copy of the DCAP Central-machine on the Backup-machine from a full backup.

1. Install DCAP Central on the Backup-machine (use [doc](https://github.com/jsonar/sonarg/blob/master/docs/sonarg_install.rst)), it is recommended to use the same DCAP Central version as the one that was used on the DCAP Central-machine. When running the DCAP Central-setup, use the same locations as on the original DCAP Central-machine.

2. Stop all the sonar services. Sonard service should be stopped last:

# sudo systemctl stop sonarfinder sonargd sonares sonarkibana sonardispatcher ...
# sudo systemctl stop sonard
  1. Move the data from the desired backup version to <sonarw home>/data/ and chown it to sonarw:sonar:

    # sudo rm -rf /var/lib/sonarw/data/*
    # sudo mv /var/lib/sonar/backup/database_backups/<Version>/* /var/lib/sonarw/data/
    # sudo chown -R sonarw:sonar /var/lib/sonarw/data/
    
  2. Run sonard to make sure there are no data related errors:

    # sudo systemctl start sonard
    
  • Don’t be alarmed if you see “Rebuilding unique id index” errors, sonarw needs to update the metadata for the new data.
  1. Restore each of the files & folders under the “configuration_backup” folder to their appropriate location, and set the permissions, based on the information in the “config_file_metadata” file located in the backup directory (default location: /var/lib/sonar/backup/configuration_backup). The metadata file contains the relevant metadata (file locations, owners, permissions, etc…).

For example, to set the sonarfinder service “config.properties” file:

# grep "config.properties" config_file_metadatasonarFinder
-rw-r--r-- 1 sonarfinder sonar 8.3K Jan  4 21:18 /opt/sonarfinder/sonarFinder/config.properties

# sudo cp /var/lib/sonar/backup/configuration_backup/config_files/20190228080001/config.properties
/opt/sonarfinder/sonarFinder/config.properties
cp: overwrite ‘/opt/sonarfinder/sonarFinder/config.properties’? y

# sudo chown sonarfinder:sonar config.properties
  1. Move all the reports from the backup directories (note: backup is only keeping the last two report versions, in the “reports” and “reports.bak folders”), to the reports folder. Then chown them to sonarfinder:sonar :

    # sudo cp /var/lib/backup/configuration_backup/reports/* <reports folder on DCAP Central-machine> # sudo chown -R sonarfinder:sonar <reports folder on DCAP Central-machine>

  2. Restart sonard and the other sonar services:

    # sudo systemctl restart sonard # sudo systemctl start sonarfinder sonargd sonares sonarkibana …

Note: The backup doesn’t back-up the users .ssh folders and keys(like the sonarw .ssh folder being set in the HADR or if there is .ssh folder for sonargd user), so in case of using them on the DCAP Central original machine, recreate them on the new machine as well.