Rsync host setup

Adding rsync backups to your backup strategy is an excellent way of insuring yourself against data loss. Critical files can be copied to a secure, offsite location, away from your office, and backing up across the internet overcomes the need to swap tapes or hard drives. You can also use built-in rsync encryption to protect the data on the rsync host, and the data can be accessed from wherever you are, using BackupAssist.

Rsync efficiently manages bandwidth and data allowances and will only transfer data that has changed. This means your internet backups will take less time when compared other remote backup methods such as FTP.

Rsync is an open source application used to synchronize files and directories from one location to another. BackupAssist’s implementation of this technology is in the form of an rsync destination that allows File Protection backup jobs to back up data across the internet. The data transfer is minimized because only the data that has changed is transmitted and all data packets are compressed.

Rsync uses a checksum method to perform the bit level data transfer. Rsync checks whether any data has changed by looking at the size of a file and its modification date. If no data has changed, rsync will not transfer the data, saving time and bandwidth. If files do not match, rsync uses a checksum method called a rolling checksum on the changed file to see what has changed. It will then transfer only the altered or appended data within the file.

Rsync terminology:

• Data Host - The remote machine that will be used as your backup destination.
• Rsync Server - The same as the data host, but specifically referring to the machine running rsync that accepts incoming connections and data from rsync clients.
• Rsync Client - A machine containing your working data that has BackupAssist installed. BackupAssist comes packaged with the rsync libraries necessary to transfer data to the rsync server during a backup.

To help better understand how rsync transfers work, let’s take a look at a hypothetical three day backup scenario.

The scenario examines three different backup methods: rsync, FTP and incremental drive imaging.

Day 1: We begin with a 4GB data file backup.

Looking at this first backup we see that for the initial data transfer there is a 100% transfer for both Incremental drive imaging and for FTP. Thanks to rsync’s packet compression we see a 50% reduction in the initial transfer. Depending on your rsync server’s setup this initial overhead can be removed by seeding your backup server locally, a method we will discuss later in this paper.

Day 2: On the second day we have added a further 0.1 GB to the start our data file.

We can see that both FTP and incremental drive imaging perform a full backup of the file. Rsync only backs up the changed data within the file, and compresses the sent data, resulting in a 50mb transfer.

Day 3: This day no data has been added, but data has been shifted within the file.

Rsync is able to recognize that the data is already on the backup server and will reorganize the file with a minimal instruction file. Incremental drive imaging is also aware that the data was moved, however it must re-backup the moved data as this section does not match the data source. FTP once again has to do a full backup of the source data.

Summary

As demonstrated in this example, rsync delivers substantial performance gains. With the ability to check what data is still the same, then append, remove or modify it as necessary to match the local source it can greatly reduce backup overhead.

The rsync destination that you use can be either an rsync server that you maintain yourself or a third party destination that supports rsync. Third party destinations include IT service providers, data centers, ISPs and cloud providers. These solutions have the advantage of high availability networks with saleable storage.

Data compression and encryption

BackupAssist supports encryption and compression on the server. BackupAssist for rsync offers industry standard encryption for data stored on the data host. This means that your data is safe “in the cloud”, making external hosting a safe and secure option. Your files are also automatically compressed on the Data Host, which reduces the amount of disk space used on your hosting company.

Show more

Rsync for BackupAssist uses four types of compression:

• Effective transfer compression by only sending changed data.
• All data packets are compressed and encrypted during transfer.
• Single Instance Store (SIS) uses hard link technology to prevent the same files from being stored more than once across backups on your host.
• The source data is encrypted and compressed in an rsync-friendly way before transmission, effectively minimizing the space used by files on the server even further.

Note: If you enable or disable encryption for an rsync job, BackupAssist will need to re-seed the backup to the host with a full set of data (i.e. the next backup will be a full backup).

Single-Instance store

File Protection backups cannot use single-instance store when the backup is saved on a ReFS formatted rsync destination. This means all the data will be backed up each time the backup job runs.

Preservation of file attributes

Because rsync works on top of the Cygwin Unix emulation layer, it does not recognize Windows file attributes (e.g. read-only, hidden), NTFS security attributes (i.e. access control lists), NTFS alternate data streams or file creation times. The only file system attribute preserved when using rsync to transfer data is the Last modified time attribute.

Show more

BackupAssist’s implementation of rsync overcomes this limitation by having the option to store NTFS metadata on the backup destination. This option is enabled in the Manage screen under Rsync options. This is checked by default for new jobs created in BackupAssist. If enabled, NTFS streams and security data will be saved to a separate file on the destination and then added back to the file as part of the restore process, when using the Integrated Restore Console. So while these attributes are not "preserved" on the files backed up to your rsync destination, they will still be restored.

This table outlines what attributes are preserved with the NTFS metadata option:

File attributes at destination	Preserved
Windows File Attributes	No
Creation time	No
Last access time	No
Last modified time	Yes
NTFS security (Access Control Lists)	Yes
NTFS alternate data streams (ADSs)	Yes

Using rsync for files and directories

Rsync performs best when working directly on the file system, backing up normal files and directories. It is able to identify which of the 50 files have changed, and for those files, it determines what changed. It calculates checksums on 50MB of data, and can complete the backup in a matter of minutes. The amount of data transferred will be around 20MB for typical documents.

Show more

Below is an example of how rsync performs:

• File system with 50,000 files, 50 GB total
• 50 files of total size 50 MB have changed

Compare this to a scenario where you use a File Protection backup job to transfer an image file created by a System Protection backup, to an rsync destination. In this example, the VHD is 50GB.

Rsync will detect that the single VHD file has changed, and needs to determine the in-file deltas. It needs to calculate checksums on 50GB of data, which may take hours. Additionally, we have found that even if the underlying file system changes very little, about 10% of a VHD file changes from day to day and needs to be transferred. So, about 5GB will be transferred.

We see here that it is greatly preferable in terms of bandwidth and CPU time the operate rsync on the underlying file system rather than a backup of that file system.

File size and the number of files

In theory, there is no limit to the number of files or directories that you can rsync. Even though rsync only transfers the data that has changed, it still must read all of the data in the file set to check what data has changed. This makes rsync internet backups a disk/CPU intensive operation that can take longer the more your data grows, no matter how little data has actually changed.

Show more

We recommend that wherever possible, you use one of the other backup methods provided in BackupAssist (such as BackupAssist’s File Archiving) to regularly archive infrequently used data, so the amount of actual data in day to day use is minimized.

We have run tests on several different file systems – a typical file system of 70,000 files and 24 GB with fewer than 50 MB of daily changes can be synced in around 10 minutes. The largest file system we’ve tested is of 200,000 files and 100 GB, which took 20 minutes to sync minimal changes.

SSH Authentication

For SSH communication, we use a public / private key method of authentication, meaning that you will only be asked for your password once (when registering with the server), and your public key will be uploaded to the server, enabling BackupAssist to log into the server in the future in a secure, password-less manner. For more information on public / private key authentication, visit the following Wikipedia article: Wikipedia Public Key Cryptography

Daemon Authentication

In Daemon mode, your password is stored in an encrypted format by BackupAssist and provided every time the backup runs. When running in Daemon mode, traffic will be unencrypted. For this reason, we recommend that you only use closed network environments, such as LANs or WANs connected by a secure VPN. VPNs inherently encrypt communication between nodes, so using rsync in Daemon mode over a VPN is still secure.

The first step is to download and install rsync on the server that will act as the rsync host server that you back up.

1. Run the cwRsyncServer installer.
2. Continue through the installation wizard, installing the package to a location of your choice.
3. During the installation, you will be presented with the popup on the right. We suggest leaving the SvcCWRSYNC account as is. Write down the password provided.
4. Click Install to install the package. Once this is finished cwRsync will be present on your system.

1. Run the CopSSH installer.
2. Continue through the installation wizard, installing the package to a location of your choice.
3. During the installation you will be presented with the popup on the right. We suggest leaving the SvcCOPSSH account as is. Write down the password provided.
4. Click Install to complete the process of installing CopSSH on your system.
5. During the Activate user part of the installation, you will be presented with a popup showing the service status and any active connections. At any time after the install you can access Activate a user from your start menu to allow SSH access to that user. You must activate at least one user before you can register an rsync client.
6. Click OK to continue your installation.

1. In the Start menu, under All Programs -> select CopSSH. The CopSSH Control Panel will open.
2. To start the process to activate a user, click on the Users tab across the top of the user interface.
3. Click on the Add button to bring up the wizard to activate a user.

Warning: Do NOT activate using your administrator account. Doing so will cause a lock down on the account due to CopSSH’s security settings. We recommend activating a newly created account.

4. Click Forward on the opening screen.
5. On the second screen, select the Domain and type in the user which you wish to activate. Click Forward once complete (admin is a manually created account we’ll use for this example).
6. Change the Access Type to Linux Shell and Sftp using the drop-down menu. Leave all Options enabled as they are by default.



7. On the fourth screen, click on Apply to complete the wizard and activate the user.

The user should now be showing as activated within the CopSSH Control Panel.

Your user’s home directory will be located at (for example) C:\Program Files\ICW\home\user.

The location of this directory can be changed by editing the file C:\Program Files\ICW\etc\passwd.

NAS hosting tips

Make sure the hardware is rsync compatible

When you select hardware to use as an rsync server, make sure the hardware can support the rsync protocol. If you select a Windows system, it must be able to run cwRsync. A NAS device must have rsync specified as one of the protocols supported. If in doubt, ask your hardware vendor for confirmation.

Processing speed is important!

Rsync can be a very processing intensive protocol - it uses checksums that calculate what data needs to be transferred. A lot of NAS devices come with lower range CPUs built-in. This will affect the overall time taken to complete an rsync backup.

Ensure there is plenty of disk space available

Although you may think you have enough disk space available when you first implement your rsync solution, a common cause of rsync problems is that the storage space eventually runs out. Some of the BackupAssist backup schemes are designed to retain significant amounts of data – meaning the space you have can be used up faster than you expect! Running out of disk space is a common problem and it can cause a lot of problems when it occurs. For this reason, the available storage space on your rsync host should be monitored.

Make sure you set the correct backup path

Some NAS devices contain a boot partition (similar to Windows Server 2008R2). Sometimes, if you enter the incorrect path your rsync backup will write to this boot partition – which could in turn cause major issues with your backup and hardware.

Seed your backup

If you’re planning on using a NAS device, you can run your seed backup by connecting your NAS device directly to the local network. This avoids having to seed to a USB drive, and then running the seed to the NAS device in a two-step process (saving you a lot of time).

Double check permissions

Even though you are logged in as a Domain Admin, most NAS devices require users to be set up locally within the unit and have permissions configured locally as well. If you receive permission issues, this is usually the reason as to why.

When using Daemon mode, a common issue is that the process is not fully automatic - unlike Rsync over SSH, you can't just specify a directory to write to and away it goes. Rsync Daemon's require a module to be set up first to make it possible to specify the backup directory.

Prerequisites

Before attempting to create a Daemon module you must make sure that cwRsync is installed on the Rsync destination. You can download a copy of cwRsync from the BackupAssist website. In addition, the steps to install cwRsync successfully are located in the sections above.

Altering the rsyncd.conf file

First, you need to create the Daemon module to be able to connect to it. This is done by altering the rsyncd.conf file, which is located in C:\Program Files (x86)\ICW by default.

Open this file in a text editor, such as Notepad, and enter something similar to the following:

[test]

path = /cygdrive/c/backup_directory

read only = false

transfer logging = yes

The meaning of each line is as follows:

[test] - This is the name of the module you're creating. (It can be set to anything you choose).

path - This is the directory that you want to back up to on the Rsync server.

read only - This sets the 'path' (see above) to either be read/write or read only. Since we want to write the backup to this directory, it should always be set to 'False'.

transfer logging - Allows any transfer information to be logged for future reference. This can be set to either Yes or No.

Once you have finished editing the rsyncd.conf file, save it to the following location: C:\Program Files (x86)\ICW\etc.

Starting the Rsync Daemon on the Rsync Destination

Next, you need to make sure that the Rsync Daemon service is started correctly. This is the service used to control the Daemon connection and if not started correctly can cause connection errors.

To start the Rsync Daemon, follow these steps:

1. Open a command prompt window by browsing to Start > Run > and typing cmd in the text field provided.
2. In the command prompt Window, change directory by typing CD "C:\Program Files\ICW\bin" or CD "C:\Program Files (x86)\ICW\bin" on 64-bit machines.
3. Start the Rsync Daemon by entering Rsync --daemon:

The Rsync Daemon should now be started and BackupAssist should be able to connect to it successfully.

Configuring BackupAssist to use the Rsync Module

The next step is to configure the BackupAssist client to perform a backup using Rsync. Install BackupAssist v5.1 or later2. You will have a free 30 day trial, but beyond this trial period, you will need to purchase a license for "BackupAssist" and "Cloud Backup add-on" to continue using it.

1. 2. 3. Select Rsync as your job type and click Next

1. To create a new BackupAssist backup job:
2. Launch the BackupAssist console and choose File protection
3. After making your backup selections, click Next
4. Select Rsync as your job type and click Next

In the Rsync Server options section (see screenshot below):

1. Enter your Rsync server name (or IP address), and choose Rsync Daemon.
2. Under Path on server, type in the name of the Rsync module you created earlier (in this example 'Test').
3. Enter the username and password that you want to connect to the Rsync server with.
4. Click the Test connection... button to test communication with the Rsync server.

If you've performed each step correctly, you should the test connection should pass and you will be on your way to performing backups using Rsync with BackupAssist!

If you have any questions about BackupAssist or setting up an Rsync backup, please contact us via email at support@backupassist.com. We'll be happy to help!

Notes

BackupAssist for Rsync is also available as a Standalone license for those wishing to perform Rsync/Internet backup only. A BackupAssist base license is not required if you are using a BackupAssist for Rsync Standalone license. You can read more about the difference between the two BackupAssist for Rsync license types here.

If you plan to purchase a new BackupAssist for Cloud Backup add-on license, you will require a BackupAssist base license for this operation. Download a free trial at www.backupassist.com. Or upgrade your existing base license at http://www.backupassist.com/purchasing/purchase.php