I have recently been forced to learn how to admin an OpenSolaris machine with ZFS filesystems. After doing some reading on ZFS, I realised how using snapshots with an rsync server would make an excellent backup solution.
Using several scripts to control rsync and the snapshots has allowed me to setup a system that has data retention periods applied and only uses the space on disk required to keep the changed files. Cloning a ZFS snapshot from a week ago (or even six months ago) allows you to create a synthetic full backup from the point in time the backup was taken.
To create my backup solution I had a spare Sun Thor x4540 storage system (the next model in line released after the Thumper storage systems). This system has 48 * 500gb SATA harddrives (with a fairly conservative disk setup gives about 16TB of usable disk space), the following solution could easily be implemented with any system that will support OpenSolaris as well as have enough disk space to fullfil the requirements.
I have created a Backup ZFS pool using all of the Thor’s 48 disks. This pools root ZFS system is mounted at /backup on the filesystem. Within this ZFS filesystem I created another filesystem /backup/rsync this will be where we will create all of the individual filesystems as our backup targets. You can do this with any pool, the important fact is having a ZFS filesystem called rsync (or whatever you wish really) and have ZFS filesystems under this point for each backup target.
root@backup-01:/# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT backup 19.0T 6.81T 12.2T 35% ONLINE - rpool 14.9G 5.49G 9.38G 36% ONLINE - root@backup-01:/# root@backup-01:/# root@backup-01:/# root@backup-01:/# zfs list NAME USED AVAIL REFER MOUNTPOINT backup 5.65T 9.90T 36.5K /backup backup/rsync 3.29T 9.90T 53.1K /backup/rsync backup/rsync/dvc 2.23T 9.90T 1.97T /backup/rsync/dvc backup/rsync/library 3.04G 9.90T 392M /backup/rsync/library backup/rsync/grp-core-cifs 451G 9.90T 436G /backup/rsync/grp-core-cifs backup/rsync/grp-web-frontend 7.56G 9.90T 7.50G /backup/rsync/grp-web-frontend backup/rsync/grp-web-mysql 632M 9.90T 254M /backup/rsync/grp-web-mysql rpool 9.83G 4.82G 81.5K /rpool rpool/ROOT 3.06G 4.82G 19K legacy rpool/ROOT/opensolaris 8.79M 4.82G 2.87G / rpool/ROOT/opensolaris-20100520-01 3.05G 4.82G 2.89G / rpool/dump 2.22G 4.82G 2.22G - rpool/export 122M 4.82G 21K /export rpool/export/home 122M 4.82G 21K /export/home rpool/export/home/smithj 122M 4.82G 122M /export/home/smithj rpool/swap 4.43G 9.15G 101M - root@backup-01:/#
Next step is creating the rsyncd.conf file for rsync and import the manifest for running the rsync service under OpenSolaris.
The rsyncd.conf file needs to be placed in /etc for rsync to automatically pickup the settings. You may use the following as a sample for your own rsyncd.conf
read only = yes list = yes uid = root gid = root [host-one] path = /backup/rsync/host-one read only = no hosts allow = <<host-one IP Address>> hosts deny = * [host-two] path = /backup/rsync/host-two read only = no hosts allow = <<host-two IP Address>> hosts deny = *
I found the original manifest for rsync at http://hub.opensolaris.org/bin/view/Community+Group+smf/manifests, then made minor modifications to work out of the box. Download the modified rsync.xml file from the end of this article and put it in the /var/svc/manifests/network folder on your OpenSolaris server. Once you have done this you need to run a service import.
root@backup-01:/# svccfg import /var/svc/manifest/network/rsync.xml root@backup-01:/# svcadm enable svc:/network/rsync:default
Then run the following to ensure there are no issues with the service
root@backup-01:/# svcs -x
When you want to add a new target for a host to backup remember to create a ZFS filesystem for that host and also add an entry into the rsyncd.conf file. Once you have updated your rsyncd.conf file you need to restart the service
root@backup-01:/# svcadm restart svc:/network/rsync:default
You can now rsync anything to your backup targets, a basic backup script for linux could be as simple as
rsync -a -v -v --progress --stats --delete --exclude "/proc" --exclude "/dev" --exclude "/sys" / backup-01::host-one
The final step is to implement a rotation/retention scheme using ZFS snapshots. The original download of snapadm.pl script came from Sun. I found a bug in this version of the script, the fixed version can be downloaded from the bottom of this article. Copy the snapadm.pl file to /usr/sbin and make the file executable.
The retention policy I have put into place is keep:
- 14 Daily Backups
- 4 Weekly Backups
- 6 Monthly Backups
To do this add the following to your root crontab
# Daily Backup - Keep 14 Days 0 12 * * * /usr/sbin/snapadm.pl -ret=14:0:0 -filesystem=backup/rsync -rec -cl=daily # Weekly Backup - Keep 4 Weeks 0 13 * * 0 /usr/sbin/snapadm.pl -ret=28:0:0 -filesystem=backup/rsync -rec -cl=weekly # Monthly Backup - Keep 6 Months 0 14 1 * * /usr/sbin/snapadm.pl -ret=182:0:0 -filesystem=backup/rsync -rec -cl=monthly
Each line contains a different backup type. Daily with a retention of 14 days. Weekly each Sunday with a retention of 28 days (4 weeks). Monthly on the first of the month with a retention of 182 days (half a year).
The Options for snapadm.pl used for the script
- -ret=Days:Hours:Minutes
- -filesystem=ZFS File System Path
- -rec – Recursively snapshot down the tree
- -cl – Class, just a way to label the backup type
The end result is the following list of snapshots
root@backup-01:~# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT backup/rsync@daily_Thu_20100520_1445 26.6K - 33.2K - backup/rsync@daily_Fri_20100521_1200 0 - 41.5K - backup/rsync@daily_Sat_20100522_1200 0 - 41.5K - backup/rsync@daily_Sun_20100523_1200 0 - 41.5K - backup/rsync@weekly_Sun_20100523_1300 0 - 41.5K - backup/rsync@daily_Mon_20100524_1200 28.2K - 46.5K - backup/rsync@daily_Tue_20100525_1200 26.6K - 48.1K - backup/rsync@daily_Wed_20100526_1200 26.6K - 48.1K - backup/rsync@daily_Thu_20100527_1200 0 - 48.1K - backup/rsync@daily_Fri_20100528_1200 0 - 48.1K - backup/rsync@daily_Sat_20100529_1200 0 - 53.1K - backup/rsync@daily_Sun_20100530_1200 0 - 53.1K - backup/rsync@weekly_Sun_20100530_1300 0 - 53.1K - backup/rsync@daily_Mon_20100531_1200 0 - 53.1K - backup/rsync@daily_Tue_20100601_1200 0 - 53.1K - backup/rsync@monthly_Tue_20100601_1400 0 - 53.1K - backup/rsync@daily_Wed_20100602_1200 0 - 53.1K - backup/rsync/dvc@daily_Tue_20100525_1200 287K - 2.06T - backup/rsync/dvc@daily_Wed_20100526_1200 5.75G - 2.07T - backup/rsync/dvc@daily_Thu_20100527_1200 26.5G - 2.02T - backup/rsync/dvc@daily_Fri_20100528_1200 17.2G - 2.00T - backup/rsync/dvc@daily_Sat_20100529_1200 3.72M - 1.97T - backup/rsync/dvc@daily_Sun_20100530_1200 0 - 2.00T - backup/rsync/dvc@weekly_Sun_20100530_1300 0 - 2.00T - backup/rsync/dvc@daily_Mon_20100531_1200 27.3G - 2.03T - backup/rsync/dvc@daily_Tue_20100601_1200 0 - 1.97T - backup/rsync/dvc@monthly_Tue_20100601_1400 0 - 1.97T - backup/rsync/dvc@daily_Wed_20100602_1200 184K - 1.96T - backup/rsync/library@daily_Fri_20100521_1200 0 - 31.5K - backup/rsync/library@daily_Sat_20100522_1200 0 - 31.5K - backup/rsync/library@daily_Sun_20100523_1200 0 - 31.5K - backup/rsync/library@weekly_Sun_20100523_1300 0 - 31.5K - backup/rsync/library@daily_Mon_20100524_1200 24.9K - 31.5K - backup/rsync/library@daily_Tue_20100525_1200 390M - 390M - backup/rsync/library@daily_Wed_20100526_1200 389M - 389M - backup/rsync/library@daily_Thu_20100527_1200 384M - 384M - backup/rsync/library@daily_Fri_20100528_1200 388M - 388M - backup/rsync/library@daily_Sat_20100529_1200 388M - 388M - backup/rsync/library@daily_Sun_20100530_1200 0 - 389M - backup/rsync/library@weekly_Sun_20100530_1300 0 - 389M - backup/rsync/library@daily_Mon_20100531_1200 391M - 391M - backup/rsync/library@daily_Tue_20100601_1200 0 - 392M - backup/rsync/library@monthly_Tue_20100601_1400 0 - 392M - backup/rsync/library@daily_Wed_20100602_1200 24.9K - 391M - backup/rsync/grp-core-cifs@daily_Fri_20100521_1200 13.7M - 301G - backup/rsync/grp-core-cifs@daily_Sat_20100522_1200 278M - 435G - backup/rsync/grp-core-cifs@daily_Sun_20100523_1200 0 - 436G - backup/rsync/grp-core-cifs@weekly_Sun_20100523_1300 0 - 436G - backup/rsync/grp-core-cifs@daily_Mon_20100524_1200 373M - 435G - backup/rsync/grp-core-cifs@daily_Tue_20100525_1200 501M - 434G - backup/rsync/grp-core-cifs@daily_Wed_20100526_1200 486M - 434G - backup/rsync/grp-core-cifs@daily_Thu_20100527_1200 492M - 434G - backup/rsync/grp-core-cifs@daily_Fri_20100528_1200 439M - 436G - backup/rsync/grp-core-cifs@daily_Sat_20100529_1200 500M - 436G - backup/rsync/grp-core-cifs@daily_Sun_20100530_1200 0 - 436G - backup/rsync/grp-core-cifs@weekly_Sun_20100530_1300 0 - 436G - backup/rsync/grp-core-cifs@daily_Mon_20100531_1200 381M - 436G - backup/rsync/grp-core-cifs@daily_Tue_20100601_1200 0 - 436G - backup/rsync/grp-core-cifs@monthly_Tue_20100601_1400 0 - 436G - backup/rsync/grp-core-cifs@daily_Wed_20100602_1200 38.1M - 437G - backup/rsync/grp-web-frontend@daily_Fri_20100521_1200 4.57M - 7.49G - backup/rsync/grp-web-frontend@daily_Sat_20100522_1200 4.95M - 7.49G - backup/rsync/grp-web-frontend@daily_Sun_20100523_1200 0 - 7.49G - backup/rsync/grp-web-frontend@weekly_Sun_20100523_1300 0 - 7.49G - backup/rsync/grp-web-frontend@daily_Mon_20100524_1200 5.44M - 7.49G - backup/rsync/grp-web-frontend@daily_Tue_20100525_1200 5.59M - 7.49G - backup/rsync/grp-web-frontend@daily_Wed_20100526_1200 5.45M - 7.50G - backup/rsync/grp-web-frontend@daily_Thu_20100527_1200 5.43M - 7.50G - backup/rsync/grp-web-frontend@daily_Fri_20100528_1200 5.57M - 7.50G - backup/rsync/grp-web-frontend@daily_Sat_20100529_1200 5.56M - 7.50G - backup/rsync/grp-web-frontend@daily_Sun_20100530_1200 0 - 7.50G - backup/rsync/grp-web-frontend@weekly_Sun_20100530_1300 0 - 7.50G - backup/rsync/grp-web-frontend@daily_Mon_20100531_1200 5.56M - 7.50G - backup/rsync/grp-web-frontend@daily_Tue_20100601_1200 0 - 7.50G - backup/rsync/grp-web-frontend@monthly_Tue_20100601_1400 0 - 7.50G - backup/rsync/grp-web-frontend@daily_Wed_20100602_1200 179K - 7.50G - backup/rsync/grp-web-mysql@daily_Thu_20100520_1445 35.7M - 257M - backup/rsync/grp-web-mysql@daily_Fri_20100521_1200 29.4M - 251M - backup/rsync/grp-web-mysql@daily_Sat_20100522_1200 29.6M - 251M - backup/rsync/grp-web-mysql@daily_Sun_20100523_1200 0 - 251M - backup/rsync/grp-web-mysql@weekly_Sun_20100523_1300 0 - 251M - backup/rsync/grp-web-mysql@daily_Mon_20100524_1200 29.8M - 251M - backup/rsync/grp-web-mysql@daily_Tue_20100525_1200 30.1M - 251M - backup/rsync/grp-web-mysql@daily_Wed_20100526_1200 30.4M - 252M - backup/rsync/grp-web-mysql@daily_Thu_20100527_1200 30.6M - 252M - backup/rsync/grp-web-mysql@daily_Fri_20100528_1200 32.9M - 254M - backup/rsync/grp-web-mysql@daily_Sat_20100529_1200 33.1M - 254M - backup/rsync/grp-web-mysql@daily_Sun_20100530_1200 0 - 254M - backup/rsync/grp-web-mysql@weekly_Sun_20100530_1300 0 - 254M - backup/rsync/grp-web-mysql@daily_Mon_20100531_1200 33.3M - 254M - backup/rsync/grp-web-mysql@daily_Tue_20100601_1200 0 - 254M - backup/rsync/grp-web-mysql@monthly_Tue_20100601_1400 0 - 254M - backup/rsync/grp-web-mysql@daily_Wed_20100602_1200 79.7K - 254M - rpool/ROOT/opensolaris-20100520-01@install 148M - 2.82G - rpool/ROOT/opensolaris-20100520-01@2010-05-19-21:49:46 13.8M - 2.87G - root@backup-01:~#
To do a file restore you need to make a ZFS Clone of the snapshot, this will allow you to copy the files required. The most important thing is that you MUST destroy the clone once you have finished the restore, failure to do so will result in the snapshot being unable to be deleted by the retention policy.
I certainly recommend reading the Solaris ZFS Administrators Guide and the ZFS Best Practices Guide. Both these guide certainly helped me understand ZFS and how to manage it.
Download:
- Rsync Manifest – rsync.xml
- Snapshot Admin Script – snapadm.pl
4 replies on “OpenSolaris Backup Solution with Rsync and ZFS Snapshots”
This is awesome, but where did you get the Thor? I’ve been dreaming of a thumper ever since I heard about their existence. Having studied the ZFS filesystem I know it is the best filesystem out there. I wish I could use it with more inexpensive solution.
Great article!
Well spotted!
:watch rsync rate over some processes (data rate after disk decompression, before network compression)
aggr_rate.d
#!/usr/sbin/dtrace -s
#pragma D option quiet
uint64_t isum;
uint64_t ilast;
dtrace:::BEGIN
{
isum = 0;
}
sysinfo::write:writech
/execname==”rsync”/
{
isum += arg0;
}
profile:::tick-1sec
{
rate = isum – ilast;
ilast = isum;
printf(“aggr bytes %20d bytes/secr”,rate);
}
:THOR DISKS…
/ on mirror c0t0d0s0 c1t0d0s0
zpool create DB mirror c2t0d0 c3t0d0
zpool create -f DATA raidz2 c0t1d0 c0t5d0 c1t2d0 c1t6d0 c2t4d0 c3t2d0 c3t6d0 c4t3d0 c4t7d0 c5t4d0
zpool add -f DATA raidz2 c0t2d0 c0t6d0 c1t3d0 c1t7d0 c2t5d0 c3t3d0 c3t7d0 c4t4d0 c5t0d0 c5t5d0
zpool add -f DATA raidz2 c0t3d0 c0t7d0 c1t4d0 c2t2d0 c2t6d0 c3t4d0 c4t0d0 c4t5d0 c5t2d0 c5t6d0
zpool add -f DATA raidz2 c0t4d0 c1t1d0 c1t5d0 c2t3d0 c2t7d0 c3t5d0 c4t2d0 c4t6d0 c5t3d0 c5t7d0
zpool add -f DATA spare c2t1d0 c3t1d0 c4t1d0 c5t1d0
zpool create BACKUPPOOL mirror c0t0d0s4 c1t0d0s4
zfs set compression=lzjb DATA
zfs set recordsize=4k|8k DB
Thanks for this post, being new to Solaris it saved me a lot of grief and heartache.
After following your instructions I found rsyncd would not start and svcs reported it as offline.
Turned out that the manifest had svc:/network/physical:default as a dependency but I had svc:/network/physical:nwam enabled instead. Changing the dependency to just svc:/network/physical fixed that for me, Diff below:
— rsync.xml_.txt 2011-08-22 18:17:28.000000000 +1000
+++ /var/svc/manifest/network/rsync.xml 2012-01-11 16:07:41.454252333 +1000
@@ -24,9 +24,9 @@
name=”physical”
grouping=”require_all”
restart_on=”error”
type=”service”>
–
+
<dependency
name="fs-local"
Thanks for this post, being new to Solaris it saved me a lot of grief and heartache.
After following your instructions I found rsyncd would not start and svcs reported it as offline.
Turned out that the manifest had svc:/network/physical:default as a dependency but I had svc:/network/physical:nwam enabled instead. Changing the dependency to just svc:/network/physical fixed that for me, Diff below:
--- rsync.xml_.txt 2011-08-22 18:17:28.000000000 +1000
+++ /var/svc/manifest/network/rsync.xml 2012-01-11 16:07:41.454252333 +1000
@@ -24,9 +24,9 @@
name="physical"
grouping="require_all"
restart_on="error"
type="service">
-
+
<dependency
name="fs-local"
repost as the form ate my diff