Hello, this is part 2 of my rSync backup solution article. Please see part 1 if you have not read it yet.
Overview
The primary weakness of the rSync bash script covered in part 1 when backing up a Windows computer is that it is unable to transfer any file that might be open by the user or operating system. Microsoft includes a backup application (NTBackup) beginning in Windows XP Service Pack 1 and Windows 2003 that gets around this problem by creating its backup from a volume shadow copy. The Volume Shadow Copy creates a snapshot of a specific volume at a specific point in time. This allows one to backup every file in a volume without the possibility of any being open or partially written to by an application.
Using the volume shadow copy requires a bit of black magic. We’ll need to obtain vshadow.exe from one of Microsoft’s freely available SDK’s. Very few programs understand the URI for a volume shadow copy, including rSync, so we’ll also need an application capable of mapping a volume shadow copy to a standard Windows drive letter. The solution is an small program called dosdev.exe, which is a wrapper around DefineDosDevice( ) Win32 API.
Setup
For our setup we will have two machines. The Mirror (192.168.1.2) will be the rSync backup server running a *nix operating system. The Mirror will create snapshots of the Target (192.168.1.3) which is running either Windows XP or Windows Vista. I will be sure to note when the two Windows operating systems differ in our setup.
Required software for the Mirror
Required software for the Target
- cwRsync
- CopSSH
- vshadow.exe
- dosdev.exe
The software needed on the Mirror is pretty minimal, assuming you already setup the snapshot-based backup covered in part 1. The one new thing we’ll need is Expect so we can write a short Expect Script.
For Windows, we’ll need a Windows based rSync. I picked cwRsync because it is easy to install and runs from Cygwin, which we’ll need. There is no need to install Cygwin, however, as cwRsync comes with all the required Cygwin binaries in the installer. You can find cwRsync at their site: http://www.itefix.no/i2/node/10650
Install cwRsync to the folder of your choice. Next, download CopSSH. CopSSH is also a straight forward to install and setup SSH server for Windows. The program is Cygwin based, and again, includes all the Cygwin binaries needed to do its job. The trick here is, we want to install CopSSH to the same location as cwRsync so we avoid having two instances of Cygwin running simultaneously.
Next, we’ll need vshadow.exe. For Windows XP and 2003 users, you can download Microsoft’s Volume Shadow Copy Service SDK. Vista user’s need to download the Windows Vista SDK Update. It is very important you obtain the correct version of vshadow.exe. For instance, the Windows XP and 2003 versions are different and will be found in different directories after the SDK is installed. If you encounter errors when running vshadow.exe, then do verify you grabbed the correct file. I suggest copying vshadow.exe and placing it in either the Cygwin bin folder or your System32 folder in Windows. You may uninstall the SDK if you prefer - we only need vshadow.exe.
Finally, we need dosdev.exe. This program is very obscure. To get the exe from Microsoft, download MPSRPT_CLUSTER.EXE. If you have a program like 7zip you can extract dosdev.exe directly from the installer. Otherwise you will need to install the report application and copy dosdev.exe from it. Again, I suggest copying dosdev.exe and placing it in either the Cygwin bin folder or your System32 folder in Windows. DO NOT EXECUTE the report…
Once everything is installed, activate a CopSSH user. This can be done easily by launching the activate user wizard from the CopSSH start menu entry. In order to execute vshadow.exe you need to activate a user with Administrator privledges. In Vista, this probably means the hidden Administrator account. Vista users, please follow this How to guide to enable the hidden Administrator account and assign it a password. If you prefer, you may then hide the account from the login screen.
Scripts
For this setup we’ll need (sadly) eight files: four on the Mirror and four more on the Target machine. Place all the Mirror files in the same directory and I have them creating the snapshot in /snapshot. For the Target, I left all four files in the Administrator’s home dir at C:\Program Files\cwRsync\home\Administrator. I plopped a copy of vshadow.exe and dosdev.exe in C:\Program Files\cwRsync\bin so that I could execute them like the Cygwin command line programs. Again, C:\Windows\System32 would also work. Replace all instances of “password123″ with your Administrator’s password. However, the rSync daemon password need not be the same as your Windows authentication password.
General idea
- backup.sh is initiated manually or from a scheduled cron entry
- backup.sh cleans up extra files in case the previous execution failed to complete
- backup.sh uses an expect script to log into Target via a password authenticated SSH session and begins vshadow.exe as a background process.
- backup.sh waits on a VB script that is started remotely to sleep until the rSyncd pid file is detected
- vshadow.exe executes a CMD script that maps a volume shadow copy with dosdev.exe and then begins a rSync daemon
- rSync daemon creates a pid file and listens for a connection
- Remote VB script detects the rSyncd pid file and terminates, allowing the bash script to continue
- backup.sh begins the rSync transfer process, which connects to the daemon running on the remote machine
- After the transfer is complete, backup.sh kills the daemon process on the remote machine
- With the rSync daemon terminated, the CMD script continues and unmaps the drive letter used for the volume shadow copy.
- vshadow.exe completes and destroys the volume shadow copy.
- backup.sh cleans up extra files and terminates.
Mirror
- backup.sh
- exclude_users
- rsync.secret
- sshlogin.exp
Target
- rsyncd.secrets
- sleep-rsyncd.vbs
- vsrsync.cmd
- rsyncd.conf
- The files backup.sh and exclude_users are recognizably close to what we used in part 1. If you are a Vista user do note that you will need to exclude certain junctions and soft links that Microsoft included to maintain XP-like paths or rSync will error and/or backup the same files more than once. The junctions might be surprising to some Windows users since Vista OS keeps them hidden from the user during normal use.
- There are some differences between my backup.sh on XP and Vista; however, it turns out the Vista version will run on XP so I will only provide my Vista script in this guide to keep things simpler.
- Notice that we call vshadow.exe and taskkill using an expect script rather than the usual public key authentication. Public key authentication in Cygwin/SSH bypasses Windows authentication (because no Windows password was provided to authenticate) and in Cygwin 1.5.x, the version I used in my setup, the context switch used to support public key authentication causes a slight difference in the environment that prevents vshadow.exe from running successfully. The win32 app taskkill also fails on Vista when used with public key authentication (it completed successfully on XP, however).
- For some reason the PID printed to the rsyncd pid file in Vista was inaccurate. In XP it was correct and I used the contents of the pid file to kill the process.
- Also take note that we’re using a rSync clinet/daemon model instead of tunneling rSync over SSH. This means we’ll need a password file to avoid an interactive password prompt and our transfer is sent over the clear instead of being encrypted. Advanced users may consider tunneling rSync over STunnel to achieve encryption.
- We set the modify window parameter to 2 seconds to the 2 second timestamp resolution of FAT32 file systems.
- Finally, while Windows paths are not case sensitive, *nix and by extension, Cygwin, paths are. On my XP machine the Windows directory is all capitals, so be sure to verify the casing in the paths I entered below match your system.
backup.sh
#!/bin/bash
unset PATH
# USER VARIABLES
BACKUPDIR=/snapshots # Folder on the backup server where the backups shall be located
KEY=/root/rsync/rsync-key # SSH key
TARGET_HOST=192.168.1.3 # The address of the production server
TARGET_USER=Administrator # The user of the production server
EXCLUDES_USERS=/root/backup/exclude_users # File containing the excluded directories
DAYS=30 # The number of days after which old backups will be deleted
# PATH VARIABLES
CSCRIPT=/cygdrive/c/Windows/System32/cscript.exe # Remote location of cscript exe
TASKKILL=/cygdrive/c/Windows/System32/taskkill.exe # Remote location of taskkill.exe
CAT=/bin/cat # Remote location of cat bin
SED=/bin/sed # Remote location of sed bin
CP=/bin/cp; # Location of the cp bin
FIND=/usr/bin/find; # Location of the find bin
ECHO=/bin/echo; # Location of the echo bin
MK=/bin/mkdir; # Location of the mk bin
SSH=/usr/bin/ssh; # Location of the ssh bin
DATE=/bin/date; # Location of the date bin
RM=/bin/rm; # Location of the rm bin
GREP=/bin/grep; # Location of the grep bin
RSYNC=/usr/bin/rsync; # Location of the rsync bin
TOUCH=/bin/touch; # Location of the touch bin
## ##
## — DO NOT EDIT BELOW THIS HERE — ##
## ##
# CREATING NECESSARY FOLDERS
$MK $BACKUPDIR
CURRENT=$BACKUPDIR/current
OLD=$BACKUPDIR/old
$MK $CURRENT
$MK $OLD
# CREATING CURRENT DATE / TIME
NOW=`$DATE ‘+%Y-%m’-%d_%H:%M`
NOW=$OLD/$NOW
$MK $NOW
# INITIALIZE VOLUME SHADOW COPY AND RSYNC DAEMON
$SSH -i $KEY $TARGET_USER@$TARGET_HOST “rm rsyncd.pid”
./sshlogin.exp $TARGET_USER@$TARGET_HOST \
“vshadow.exe -script=vss-setvar.cmd -exec=vsrsync.cmd C\:” &
# SLEEP UNTIL RSYNCD PID FILE IS FOUND
$SSH -i $KEY $TARGET_USER@$TARGET_HOST “$CSCRIPT sleep-rsyncd.vbs”
# RUN RSYNC INTO CURRENT
$RSYNC \
-apvz –delete –delete-excluded \
–exclude-from=”$EXCLUDES_USERS” \
–password-file=rsync.secret \
–modify-window=2 \
$TARGET_USER@$TARGET_HOST::C/Users/ \
$CURRENT ;
# TERMINATE RSYNC DAEMON AND VOLUME SHADOW COPY
./sshlogin.exp $TARGET_USER@$TARGET_HOST “$TASKKILL -f -im rsync.exe”
$SSH -i $KEY $TARGET_USER@$TARGET_HOST “rm rsyncd.pid vss-setvar.cmd”
# UPDATE THE MTIME TO REFELCT THE SNAPSHOT TIME
$TOUCH $BACKUPDIR/current
# MAKE HARDLINK COPY
$CP -al $CURRENT/* $NOW
# REMOVE OLD BACKUPS
for FILE in “$( $FIND $OLD -maxdepth 1 -type d -mtime +$DAYS )”
do
$RM -Rf $FILE
# $ECHO $FILE
done
exit 0
sshlogin
#!/usr/bin/expect -f
# set Variables
# set password [lindex $argv 0]
set host [lindex $argv 0]
set scriptname [lindex $argv 1]
set timeout -1
# now connect to remote UNIX box (ipaddr) with given script to execute
spawn ssh -p 22 $host $scriptname
match_max 100000
expect {
-re “.*Are.*.*yes.*no.*” {
send “yes\n”
expect {
“>” { }
“$” { }
}
exp_continue
#look for the password prompt
}
“assword:” {
send — “password123\n”
expect {
“>” { }
“$” { }
}
#the expect command will now return
}
default {
send_user “Login failed\n”
exit
}
}
sleep 2
send_user “Finished\n”
exit
rsync.secret
password123
- For the Target machine we’ll need a CMD and VB Script, as well as a configuration file for the rSync daemon.
- Volume shadow copies in XP and Vista are temporary and only exist as long as the process that created it exists. In our case the process that creates the volume shadow copy is vshadow.exe (Windows 2003 users can create a persistent volume shadow copy). Fortunately, vshadow.exe allows us to send it a CMD script to execute - hence the purpose of vsrsync.cmd below. vsrsync.cmd executes after the volume shadow copy is created but before it is destroyed. During this time we need to use dosdev to map the volume shadow copy to a standard Windows drive letter that rSync can read from. Next, it will create the rSync daemon using a configuration file and secrets file. And finally, after the remote script kills the rSync daemon process, the CMD script will unmap the volume shadow copy drive letter.
- Because we need to initiate the rSync transfer after the volume shadow copy is created but before it is destroyed, we need a small VB Script that halts the remote script from executing until it detects the daemon is ready to begin the transfer. It performs this check by watching for the rSyncd pid file.
rsyncd.secrets
Administrator:password123
sleep-rsyncd.vbs
Const Flag = "C:\Program Files\cwRsync\home\Administrator\rsyncd.pid"
While DoesFileExist(Flag)=0
wscript.sleep 10000
Wend
function DoesFileExist(FilePath)
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
if not fso.FileExists(FilePath) then
DoesFileExist = 0
else
DoesFileExist = 1
end if
Set fso = Nothing
end function
vsrsync.cmd
@ECHO OFF
call vss-setvar.cmd
SET PATH=%PATH%;”C:\Program Files\cwRsync\bin”
dosdev X: %SHADOW_DEVICE_1%
rsync –daemon –config=rsyncd.conf –no-detach
dosdev -r -d X:
rsyncd.conf
use chroot = false
strict modes = false
pid file = rsyncd.pid
[C]
path = /cygdrive/X/
read only = yes
ignore nonreadable = yes
hosts allow = 192.168.1.2
auth users = Administrator
secrets file = rsyncd.secrets
I hope this guide provided some help to anyone wanting to setup something similar. My setup is largely based on the one covered at http://www.goodjobsucking.com/?p=62
Additional notes
- If you have files with special characters in the file name, you may need to upgrade your Cygwin DLL to the Unicode aware version.
- There are a lot of moving parts in the above script. If something hangs or you just need to abort and try again you may need to reboot the Windows machine to ensure all the pieces are ready for another trial. In my experience, however, killing the rSync process from Task Manager was frequently enough to bring the entire script back to a state where I could try another test run.
- cwRsync will simply abort execution if the pid file already exists. My script has an extra delete before vshadow.exe is called to try and avoid this problem.
- vshadow.exe will fail on XP and Vista if another instance of it is already running on the same volume.
Unknown8063 Inside BluHelix, Technology