VM Server Backup Strategy
From MyWiki
Contents |
Preamble
There are excellent Tivoli instructions and information at this page:
http://setwiki.ncsa.uiuc.edu/wiki/index.php/TivoliClientInstall
Daily Backups
Daily backups are accomplished on selected VMs by installing Tivoli inside the Guest O/S and configuring and running it in the normal way.
Not all VMs are backed up daily using Tivoli; see VM Servers for the current list. However, every running VM is backed up each night (and hence the previous version over-written) on its respective host under /var/vmbackups/hostname/ . The backups are tar gzipped files.
Checking the Date of the Last Tivoli Backup
/usr/bin/dsmc q fi
More info about Tivoli is here:
http://setwiki.ncsa.uiuc.edu/wiki/index.php/TivoliClientFAQ
and here:
http://setwiki.ncsa.uiuc.edu/wiki/index.php/Main_Page#Backups
You can also:
Search by machine name (or any part of a machine name):
http://backup.ncsa.uiuc.edu/query/node/machinename
Search by user name (or any part of a user name):
http://backup.ncsa.uiuc.edu/query/user/username
The above are useful for bookmarks to show all machines matching the search name.
Initialization and Cron Processing
Most Servers
Most linux servers should use the rc-style init processing to manage the scheduler process, i.e.:
service tsm start
Instructions for how to install this script is found here:
http://setwiki.ncsa.uiuc.edu/wiki/index.php/TivoliClientInstall#Install_Redhat_rc-script
Other Servers
- Add the following to /etc/rc.d/rc.local (this was in /etc/inittab, but moved to rc.local with the use of upstart [Fedora 9 and later]):
# TSM Client Scheduler/Web Startup /opt/tivoli/tsm/client/ba/bin/dsmcad nohup /opt/tivoli/tsm/client/ba/bin/dsmc sched > /dev/null 2>&1 &
All Servers
- Add a firewall rule to allow incoming connections from the Tivoli server:
-A RH-Firewall-1-INPUT -s 141.142.15.0/24 -p tcp -m state --state NEW -m tcp --dport 1501 -j ACCEPT
- Add a cronjob to restart tsm if it stopped for any reason:
#------------------------------------------------------------ # Tivoli Scheduler 30 2 * * * /etc/rc.d/init.d/tsm start >/dev/null 2>&1 #------------------------------------------------------------
MySQL Backup Processing
30 1 * * * $HOME/wikidb_backups/database_backup.sh
- The database_backup.sh script ensures a clean text-based backup of the MySQL database. The contents of /root/wikidb_backups/database_backup.sh are:
#!/bin/bash date=`date +D%Y%m%dT%H%M%S` outfile=/root/wikidb_backups/backup$date.sql mysqldump -u wikiuser -pXXXXX --opt wikidb >$outfile
Weekly Backups
All VMs are backed up weekly.
This is accomplished as follows:
Cron Processing
On Saturday mornings at 2AM, a cron job starts that performs the following actions:
- Loop through all Guest VMs, one at a time, doing the following:
- Suspend the guest machine
- Tar up and compress the guest machine files to /var/vmbackups/<machinename>.tar.gz
- Startup the guest machine
The crontab Entry
[root@pkirack8 ~]# crontab -l #------------------------------------------------------------ # VMware server status change check. Run hourly 0 * * * * /root/status.pl #------------------------------------------------------------ #------------------------------------------------------------ # Tivoli Scheduler 30 8 * * * /opt/tivoli/tsm/client/ba/bin/dsm_sch #------------------------------------------------------------ #------------------------------------------------------------ # System update. Run daily at midnight 0 0 * * * /usr/sbin/up2date -u #------------------------------------------------------------ #------------------------------------------------------------ # VM Backups. Run daily at 11pm 0 23 * * * /root/vm_backup.pl #------------------------------------------------------------
The Backup Script
The backup script, written in Perl, can be found here
The Status Script
The status script, written in Perl, can be found here
Tivoli Processing
On Saturdays at 8AM, the Tivoli client starts and backs up the host machine in the normal way.
Tivoli starts the backup window at 10pm every night and the window remains open till 11am the next morning. All scheduled nodes will start backup at a random time between 10pm - 4:30am.
Notes:
- The Tivoli client is configured (in the /opt/tivoli/tsm/client/ba/bin/dsm.sys file) to exclude the /var/lib/vmware/Virtual Machines directory
Software Download and Installation
Software and installation instructions for the Tivoli client can be found at
http://backup.ncsa.uiuc.edu/setup/
Restoring a VM
The time-granularity on restores is from the previous night (~2am) or any previous week (every Saturday ~8am). Users requiring more frequent backup should set up Tivoli internally on their VM, or use svn.ncsa.uiuc.edu or another backup method.
Restoring the Previous Day's Backup
- Untar and uncompress the correct VM from the previous night's backup:
mkdir /var/lib/vmware/Virtual\ Machines/hostname2 cd /var/lib/vmware/Virtual\ Machines/hostname2 tar xvfz /var/vmbackups/hostname # the tar might create an extra level of directory, not sure how the archive was created.
- Use the vmware console and power down the bad VM.
- Swap the directory names and restart the VM from the vmwar console.
- Delete the bad VM directory after the VM restarts.
Restoring a VM from an Older Tivoli Backup
- On the host machine, use the Tivoli command line to retrieve the previous version of the VM files, e.g.:
dsmc restore -pitd=1/23/2009 "/var/lib/vmware/Virtual Machines/vmName/" /tmp/
pitd is the prior backup date. More information can be found at: http://backup.ncsa.uiuc.edu/tsm/
- Once the files are restored, approximately follow the instructions in the section immediately above Restoring the Previous Day's Backup
Windows XP Backup Timeout Error
/var/lib/vmware/Virtual\ Machines/cctt-winxp/Windows\ XP\ Professional.vmx VMControl error -8: Invalid operation for virtual machine's current state: The requested operation ("suspend") could not be completed because it conflicted with the state of the virtual machine ("suspended") at the time the request was received. This error often occurs because the state of the virtual machine changed before it received the request. API Version: 1.01
Solution: Disable the screen saver and the monitor timeout inside of the Windows XP guest, then the command line suspend works properly from the /root/vm_backup.pl script.
http://communities.vmware.com/message/748601;jsessionid=0A4E85B61F223953D613A3F46D32A101
http://communities.vmware.com/thread/5580
