Backup with Amanda: tape, NAS, Amazon S3
From MyWiki
Contents |
Amanda
Preparing the server
Syslog settings
We need to configure Syslog to send messages out to Logstash through Redis in RELP format.
root@amdbkp-01:~# apt-get update root@amdbkp-01:~# apt-get install python-software-properties root@amdbkp-01:~# add-apt-repository ppa:adiscon/v8-stable You are about to add the following PPA to your system: Contains the latest RSyslog V8-Stable packages and dependencies. More info: https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable Press [ENTER] to continue or ctrl-c to cancel adding it gpg: keyring `/tmp/tmpLRDNaP/secring.gpg' created gpg: keyring `/tmp/tmpLRDNaP/pubring.gpg' created gpg: requesting key 5234BF2B from hkp server keyserver.ubuntu.com gpg: /tmp/tmpLRDNaP/trustdb.gpg: trustdb created gpg: key 5234BF2B: public key "Launchpad PPA for Adiscon" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK
Another apt-get update
root@amdbkp-01:~# apt-get update
Install RELP library and rsyslog
root@amdbkp-01:~# apt-get install librelp0 rsyslog rsyslog-relp
Create configuration file to send messages to Redis in RELP format
root@amdbkp-01:~# vi /etc/rsyslog.d/sendsyslog2logstash.conf
Add this line into the file, save and exit.
action(type="omrelp" target="redis-m.production" port="2514")
Add RELP configuration to /etc/rsyslog.conf file (add the below lines into Modules section):
#RELP Output module(load="omrelp")
Turn off repeated message reduction, so you have every message instead of those pesky "message repeated N times" lines.
# Filter duplicated messages $RepeatedMsgReduction off
Restart Syslog
root@amdbkp-01:~# service rsyslog restart
Sysctl settings
Some of the estimates from clients may run for some time (tens of minutes) and TCP connection might be killed due to inactivity by Fortigate firewall. Turns out Fortigates have a default setting of five minutes (see the article). Refer to Connection reset by peer article for more details.
We need to reduce net.ipv4.tcp_keepalive_time from default 7200 to 180 to survive that.
Put the below into /etc/sysctl.conf:
# Amanda related, please, do not remove! # Ref http://wiki.zmanda.com/index.php/Mesg_read:_Connection_reset_by_peer net.ipv4.tcp_keepalive_time = 180
Change it while the server is still running:
root@amdbkp-01:~# echo 180 > /proc/sys/net/ipv4/tcp_keepalive_time
Installing the software
- WARNING* Restore test revealed problem with the Amanda 3.3.0 included by default with Ubuntu 12.04. Ended up installing the latest version of Amanda 3.3.5 downloaded directly from Amanda.
Thanks to this thread for the hint!
First we pull packages from Amanda site and remove default Ubuntu 12.04 LTS Amanda packages (if installed).
root@amdbkp-01:~# cd /tmp
root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb .
root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb .
root@amdbkp-01:~# apt-get remove amanda-server amanda-client amanda-common
This packages depend on xinetd and get text been installed. Since we are not using xinetd, we will ignore it, but install gettext
Inetd way
If you want to stay with inetd. Mind you, it will create problem with 'apt-get install' later.
root@amdbkp-01:~# apt-get will be throwing this error later root@amdbkp-01:~# apt-get install gettext root@amdbkp-01:~# dpkg -i --ignore-depends=xinetd ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb
Put back the line to inetd.conf. You don't really need this until you want to stick with inetd.
root@amdbkp-01:~# tail -1 /etc/inetd.conf amanda stream tcp nowait amandabackup /usr/libexec/amanda/amandad amandad -auth=bsdtcp amdump amindexd amidxtaped
Kick inetd to reload its config
root@amdbkp-01:~# ps -ef | grep inetd root@amdbkp-01:~# kill -HUP 28822
Check that inetd picked up the config
root@amdbkp-01:~# netstat -anp | grep 1008 tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 28822/inetd
Xinetd way
The packages built by Amanda team are dependant on xinet and unless you like some pain, you better stick to the defaults
root@amdbkp-01:~# apt-get install gettext xinetd root@amdbkp-01:~# dpkg -i ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb
Fix permissions on the directories that were owned by Ubuntu's own Amanda package backup:backup
root@amdbkp-01:~# chown -R amandabackup:disk /var/lib/amanda root@amdbkp-01:~# chown -R amandabackup:disk /etc/amanda root@amdbkp-01:~# chown -R amandabackup:disk /tmp/amanda root@amdbkp-01:~# chown -R amandabackup:disk /amanda
Amanda 3.3.5 .amandahosts is not /etc ---> /var/lib/amanda/.amandahosts
amandabackup@amdbkp-01:~$ ls -l /var/lib/amanda/.amandahosts -rw------- 1 amandabackup disk 250 Jul 1 01:14 /var/lib/amanda/.amandahosts amandabackup@amdbkp-01:~$ cat /var/lib/amanda/.amandahosts localhost root amindexd amidxtaped localhost.localdomain root amindexd amidxtaped localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump amdbkp-01 amandabackup amdump
Time
Fix /etc/ntp.conf to point to internal NTP servers 192.168.0.10 and 192.168.1.10
driftfile /var/lib/ntp/ntp.drift statistics loopstats peerstats clockstats filegen loopstats file loopstats type day enable filegen peerstats file peerstats type day enable filegen clockstats file clockstats type day enable server 192.168.1.10 server 192.168.0.10 server ntp.ubuntu.com restrict -4 default kod notrap nomodify nopeer noquery restrict -6 default kod notrap nomodify nopeer noquery restrict 127.0.0.1 restrict ::1
ntpq> pe
remote refid st t when poll reach delay offset jitter
==============================================================================
*192.168.1.10 193.1.219.116 2 u 13 64 37 0.169 10.502 1.580
192.168.0.10 91.189.94.4 3 u 11 64 37 0.119 5.908 1.268
91.189.94.4 193.79.237.14 2 u 13 64 37 13.113 5.955 1.247
Holding space
Create logical volume and filesystem:
root@amdbkp-01:~# lvcreate -n amandalv -L100G amdbkp-01-vg root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/amandalv root@amdbkp-01:~# mount /dev/amdbkp-01-vg/amandalv /amanda root@amdbkp-01:~# chown amandabackup:disk /amanda root@amdbkp-01:~# chmod g+w /amanda
Add the new filesystem to /etc/fstab
root@amdbkp-01:~# grep amanda /etc/fstab /dev/mapper/amdbkp--01--vg-amandalv /amanda ext4 defaults 0 2
Postfix configuration
We only need to send emails out, not to receive any, so the configuration is as slim as it can be:
root@amdbkp-01:~# cat /etc/postfix/main.cf relayhost = smtp.production transport_maps = hash:/etc/postfix/transport root@amdbkp-01:~# cat /etc/postfix/transport production smtp:smtp.production corporate smtp:smtp.production root@amdbkp-01:~# cat /etc/aliases # See man 5 aliases for format postmaster: root root: alex.markelov @corporate
Start Postfix
root@amdbkp-01:~# service postfix start
Configuring encryption
Create encryption key
root@amdbkp-01:~# pwgen -s 45 -N1
Save the key in /var/lib/amanda/.am_passphrase
root@amdbkp-01:~# vi /var/lib/amanda/.am_passphrase root@amdbkp-01:~# chown amandabackup:disk /var/lib/amanda/.am_passphrase root@amdbkp-01:~# chmod 0600 /var/lib/amanda/.am_passphrase root@amdbkp-01:~# ls -l ~amandabackup/.am_passphrase -rw------- 1 amandabackup disk 77 Jul 1 00:46 /var/lib/amanda/.am_passphrase
Dumptype tar-client-encrypt is responsible for encrypted backups. An example below makes full backups every time.
define dumptype tar-client-encrypt {
always-full-tar
comment "partitions dumped with tar + encrypted"
encrypt client
client_encrypt "/usr/sbin/amcrypt-ossl"
client_decrypt_option "-d"
}
Wrapper script to report backup status to Logstash/Kibana
Below is the script that engages amdump for given configuration and then sends report in JSON format to Redis.
#
# Wrapper script to run dump and pump report out to Logstash/Kibana
#
# Pre-requisites: mlogger, RELP configuration in rsyslog
#
PATH=/usr/sbin:/usr/bin:/bin
LOG_PRG_NAME=amanda_report
if [ $# -lt 1 ]
then
echo "Usage: `basename $0` <backup_configuration_name>"
exit -1
fi
# Perform the backup
amdump $1
# send out JSON formatted report to Logstash
amreport --format=json, $1 | sed -e 's/^$/ /g' | tr -d '\n' | mlogger -t ${LOG_PRG_NAME}
When you create cron job for backing up, use the script (example below)
# Data Integration backups 0 20 * * * /etc/amanda/amdump_and_report.sh prod-1week-vtape-RAIT
Tape library
Changer is available as /dev/sg9 device:
root@amdbkp-01:~# mtx -f /dev/sg9 status
Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export )
Data Transfer Element 0:Empty
Storage Element 1:Full
Storage Element 2:Full
Storage Element 3:Full
Storage Element 4:Full
Storage Element 5:Empty
Storage Element 6:Empty
Storage Element 7:Empty
Storage Element 8:Empty
Storage Element 9:Empty
Storage Element 10:Empty
Storage Element 11:Empty
Storage Element 12:Empty
Storage Element 13:Empty
Storage Element 14:Empty
Storage Element 15:Empty
Storage Element 16:Empty
You can load particular tape using mtx command:
root@amdbkp-01:~# mtx -f /dev/sg9 load 1
Loading media from Storage Element 1 into drive 0...done
root@amdbkp-01:~# mtx -f /dev/sg9 status
Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded)
Storage Element 1:Empty
Storage Element 2:Full
Storage Element 3:Full
Storage Element 4:Full
Storage Element 5:Empty
Storage Element 6:Empty
Storage Element 7:Empty
Storage Element 8:Empty
Storage Element 9:Empty
Storage Element 10:Empty
Storage Element 11:Empty
Storage Element 12:Empty
Storage Element 13:Empty
Storage Element 14:Empty
Storage Element 15:Empty
Storage Element 16:Empty
Checking tape status using mt command:
root@amdbkp-01:~# mt -f /dev/nst0 status drive type = 114 drive status = 1140850688 sense key error = 0 residue count = 0 file number = 0 block number = 0
Preparing tapes
Test backup
Test restore
Test encryption
Amazon S3
Reference http://wiki.zmanda.com/index.php/How_To:Backup_to_Amazon_S3
Levente created Access KeyID and Secret Access Key for me
Logged in to S3 console and created bucket called infra there.
In the bucket created folder called Amanda where the vTapes will reside.
Add the server itself to allowed hosts
root@amdbkp-01:~# cat /etc/amandahosts localhost amandabackup localhost root amindexd amidxtaped amdbkp-01 amandabackup amdbkp-01.production amandabackup
Create production-S3 config
root@amdbkp-01:~# su - amandabackup amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3 amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/log amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/index
Below is content of configuration file /etc/amanda/production-S3/amanda.conf
- WARNING* S3 secret key is not the real one in the config.
org " production" # your organization name for reports
mailto "root alex.markelov @corporate" # space separated list of operators at your site
dumpuser "amandabackup" # the user to run dumps under
inparallel 4 # maximum dumpers that will run in parallel (max 63)
# this maximum can be increased at compile-time,
# modifying MAX_DUMPERS in server-src/driverio.h
dumporder "sssS" # specify the priority order of each dumper
# s -> smallest size
# S -> biggest size
# t -> smallest time
# T -> biggest time
# b -> smallest bandwitdh
# B -> biggest bandwitdh
# try "BTBTBTBTBTBT" if you are not holding
# disk constrained
taperalgo first # The algorithm used to choose which dump image to send
# to the taper.
# Possible values:
# [first|firstfit|largest|largestfit|smallest|last]
# Default: first.
# first First in - first out.
# firstfit The first dump image that will fit on
# the current tape.
# largest The largest dump image.
# largestfit The largest dump image that will fit on
# the current tape.
# smallest The smallest dump image.
# last Last in - first out.
displayunit "k" # Possible values: "k|m|g|t"
# Default: k.
# The unit used to print many numbers.
# k=kilo, m=mega, g=giga, t=tera
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec
dumpcycle 1 weeks # the number of days in the normal dump cycle
runspercycle 5 # the number of amdump runs in dumpcycle days
# (4 weeks * 5 amdump runs per week -- just weekdays)
tapecycle 15 tapes # the number of tapes in rotation
# 4 weeks (dumpcycle) times 5 tapes per week (just
# the weekdays) plus a few to handle errors that
# need amflush and so we do not overwrite the full
# backups performed at the beginning of the previous
# cycle
usetimestamps yes # Default: Yes. This option allows Amanda to track
# multiple runs per calendar day. The only reason one
# might disable it is that Amanda versions before 2.5.1
# can't read logfiles written when this option was
# enabled.
bumpsize 20 Mb # minimum savings (threshold) to bump level 1 -> 2
bumppercent 20 # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1 # minimum days at each level
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
runtapes 1 # number of tapes to be used in a single run of amdump
# amazonaws S3
device_property "S3_BUCKET_LOCATION" "eu-west-1" # Your S3 bucket location
device_property "S3_ACCESS_KEY" "XXX" # Your S3 Access Key
device_property "S3_SECRET_KEY" "XXX" # Your S3 Secret Key
device_property "S3_SSL" "YES" # Curl needs to have S3 Certification Authority (Verisign today)
# in its CA list. If connection fails, try setting this no NO
tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09,10}" # Number of tapes in your "tapecycle"
changerfile "s3-statefile" # Amanda will create this file
tapetype S3
define tapetype S3 {
comment "S3 Bucket"
length 30 gigabytes # Bucket size 30GB
}
maxdumpsize -1 # Maximum number of bytes the planner will schedule
# for a run (default: runtapes * tape_length).
labelstr "^PRODUCTION-[0-9][0-9]*$" # label constraint regex: all tapes must match
holdingdisk hd1 {
comment "main holding disk"
directory "/amanda" # where the holding disk is
use -100 Mb # how much space can we use on it
# a non-positive value means:
# use all space but that value
chunksize 1Gb # size of chunk if you want big dump to be
# dumped on multiple files on holding disks
# N Kb/Mb/Gb split images in chunks of size N
# The maximum value should be
# (MAX_FILE_SIZE - 1Mb)
# 0 same as INT_MAX bytes
}
autoflush yes #
# if autoflush is set to yes, then amdump will schedule all dump on
# holding disks to be flush to tape during the run.
infofile "/etc/amanda/production-S3/curinfo" # database DIRECTORY
logdir "/etc/amanda/production-S3/log" # log directory
indexdir "/etc/amanda/production-S3/index" # index directory
define dumptype global {
comment "Global definitions"
# This is quite useful for setting global parameters, so you don't have
# to type them everywhere. All dumptype definitions in this sample file
# do include these definitions, either directly or indirectly.
# There's nothing special about the name `global'; if you create any
# dumptype that does not contain the word `global' or the name of any
# other dumptype that contains it, these definitions won't apply.
# Note that these definitions may be overridden in other
# dumptypes, if the redefinitions appear *after* the `global'
# dumptype name.
# You may want to use this for globally enabling or disabling
# indexing, recording, etc. Some examples:
# index yes
# record no
# split_diskbuffer "/raid/amanda"
# fallback_splitsize 64m
}
define dumptype always-full {
global
comment "Full dump of this filesystem always"
compress none
priority high
dumpcycle 0
exclude list ".exclude.lst"
}
define dumptype always-full-tar {
global
program "GNUTAR"
comment "Full backup with tar of this filesystem always"
compress none
priority high
index
dumpcycle 0
holdingdisk required
exclude list ".exclude.lst"
}
define dumptype tar-client-encrypt {
always-full-tar
comment "partitions dumped with tar + encrypted"
encrypt client
client_encrypt "/usr/sbin/amcrypt-ossl"
client_decrypt_option "-d"
}
define interface local {
comment "a local disk"
use 1000 kbps
}
define interface le0 {
comment "10 Mbps ethernet"
use 400 kbps
}
Creating vTapes
Create first 9 vTapes in S3 and label it
amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done;
Check slot-09 vTape
amandabackup@amdbkp-01:~$ amdevcheck production-S3 s3:infra/Amanda/production-S3/slot-09
The command should return SUCCESS if everything is OK
Test backup
Add /etc to disklist for test purpose:
amandabackup@amdbkp-01:~$ cat /etc/amanda/production-S3/disklist amdbkp-01 /etc tar-client-encrypt
Back it up:
amandabackup@amdbkp-01:~$ amdump production-S3 &
[1] 28522
backup@amdbkp-01:~$ amstatus production-S3
Using /etc/amanda/production-S3/log/amdump.1
From Mon Jun 30 23:58:09 IST 2014
amdbkp-01:/etc 0 3251k finished (23:58:11)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 3240k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 3251k 3240k (100.34%) (100.34%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 1 3251k 3240k (100.34%) (100.34%)
tape 1 : 1 3251k 3240k ( 0.01%) PRODUCTION-001 (2 chunks)
4 dumpers idle : no-dumpers
taper status: Idle
taper qlen: 0
network free kps: 600
holding space : 97673216k (100.00%)
chunker0 busy : 0:00:00 ( 14.23%)
dumper0 busy : 0:00:00 ( 13.83%)
taper busy : 0:00:01 ( 42.72%)
0 dumpers busy : 0:00:02 ( 86.13%) not-idle: 0:00:01 ( 50.26%)
no-dumpers: 0:00:01 ( 49.74%)
1 dumper busy : 0:00:00 ( 13.87%)
[1]+ Done amdump production-S3
Check S3 web UI to see that the tape PRODUCTION-001 was used
Test restore
You need to make sure your /etc/amanda/amanda-client.conf doesn't have tapedev defined. It led to error for S3 when I was trying to set it to what I believed was correct (tapedev "s3:infra/Amanda" or tapedev "s3:infra/Amanda/production-S3").
Error I was getting "Amanda header not found -- unlabeled volume?"
root@amdbkp-01:~# amrecover production-S3 AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-01) 200 Working date set to 2014-07-01. 200 Config set to production-S3. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> setdisk /etc 200 Disk set to /etc. amrecover> lcd /tmp amrecover> add inetd.conf Added file /inetd.conf amrecover> extract Extracting files using tape drive s3:infra/Amanda on host localhost. The following tapes are needed: PRODUCTION-002 Extracting files using tape drive s3:infra/Amanda on host localhost. Load tape PRODUCTION-002 now Continue [?/Y/n/s/d]? y Amanda header not found -- unlabeled volume? Load tape PRODUCTION-002 now Continue [?/Y/n/d]? n Got no header and data from server, check in amidxtaped.*.debug and amandad.*.debug files on server amrecover> quit 200 Good bye.
Commenting out tapedev in amanda-client.conf solved the issue. Ref: https://forums.zmanda.com/showthread.php?2811-Problem-amrecover-from-S3
root@amdbkp-01:/tmp/1# cat /etc/amanda/amanda-client.conf # # amanda-client.conf - sample Amanda client configuration file. # # This file normally goes in /etc/amanda/amanda-client.conf. # conf "DailySet1" # your config name index_server "localhost" # your amindexd server tape_server "localhost" # your amidxtaped server #tapedev "s3:infra/Amanda" #tapedev "tape:/dev/YOUR-TAPE-DEVICE-HERE" # your tape device # if not set, Use configure or ask server. # if set to empty string "", ask server # amrecover will use the changer if set to the value # of 'amrecover_changer' in the server amanda.conf. # auth - authentication scheme to use between server and client. # Valid values are "bsd", "bsdudp", "bsdtcp", "krb5", "local", # "rsh" and "ssh". # Default: [auth "bsdtcp"] auth "bsdtcp" ssh_keys "" # your ssh keys file if you use ssh auth
Successful restore after fixing /etc/amanda/amanda-client.conf. We will create /tmp/1 to restore file into.
root@amdbkp-01:/tmp# mkdir /tmp/1
root@amdbkp-01:/tmp# amrecover production-S3
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-01)
200 Working date set to 2014-07-01.
200 Config set to production-S3.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> lcd /tmp/1
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> add inetd.conf
Added file /inetd.conf
amrecover> extract
Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost.
The following tapes are needed: PRODUCTION-002
Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost.
Load tape PRODUCTION-002 now
Continue [?/Y/n/s/d]? y
Restoring files into directory /tmp/1
All existing files in /tmp/1 can be deleted
Continue [?/Y/n]? y
./inetd.conf
amrecover> quit
200 Good bye.
Now checking the file we just restored:
root@amdbkp-01:/tmp# cd 1 root@amdbkp-01:/tmp/1# ls -al total 12 drwxr-xr-x 2 root root 4096 Jul 1 01:45 . drwxrwxrwt 6 root root 4096 Jul 1 01:45 .. -rw-r--r-- 1 root root 1170 Jul 1 01:12 inetd.conf
Test encryption
- NOTE* You can use the same method to download vTape from S3 and restore locally without using Amanda
To prove that we have the vTapes encrypted at S3, first, download vTape from S3 using web UI
Alexs-MacBook:Downloads alex$ ls -l slot-02f00000001-b0000000000000000.data -rw-r--r--@ 1 alex staff 3512368 1 Jul 01:16 slot-02f00000001-b0000000000000000.data
Copy the file over to Amanda server
Alexs-MacBook:Downloads alex$ scp slot-02f00000001-b0000000000000000.data amdbkp-01:/tmp/ slot-02f00000001-b0000000000000000.data 100% 3430KB 3.4MB/s 00:01
Check the vTape with and without decryption
root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |/usr/sbin/amcrypt-ossl -d|tar tvf - drwxr-xr-x root/root 1865 2014-07-01 01:13 ./ drwxr-xr-x root/root 6 2014-04-24 15:25 ./X11/ drwxr-xr-x root/root 1 2012-07-03 07:21 ./X11/xkb/ drwxr-xr-x root/root 22 2014-04-24 15:32 ./acpi/ drwxr-xr-x root/root 11 2014-04-24 15:32 ./acpi/events/ drwxr-xr-x root/root 952 2014-06-11 06:36 ./alternatives/ drwxrwx--- amandabackup/disk 53 2014-07-01 00:46 ./amanda/ <more output skipped>
Now, try without decryption part of the command (/usr/sbin/amcrypt-ossl -d) and see how tar can't do anything with the tape:
root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |tar tvf - tar: This does not look like a tar archive tar: Skipping to next header 107+1 records in 107+1 records out tar: 3512368 bytes (3.5 MB) copiedExiting with failure status due to previous errors, 0.00662878 s, 530 MB/s
NAS based vTapes
The below based on vTape setup in Amanda wiki.
We need NFS client
root@amdbkp-01:~# apt-get install nfs-common
Mount the test NFS share and prep directory for vTapes
root@amdbkp-01:~# mount.nfs dc-nas-01:/test /vtapes root@amdbkp-01:~# mkdir /vtapes/amanda.vtapes root@amdbkp-01:~# chown amandabackup:disk /vtapes/amanda.vtapes root@amdbkp-01:~# ls -dl /vtapes/amanda.vtapes drwxrwx---+ 2 amandabackup disk 4096 Jul 7 10:50 /vtapes/amanda.vtapes
Creating vTapes
Switch to amanda user
root@amdbkp-01:~# su - amandabackup
Prepare configuration directory. We will take S3 directory and files as the starting point:
amandabackup@amdbkp-01:~$ cd /etc/amanda/ amandabackup@amdbkp-01:~$ cp -rp production-S3 production-vtape amandabackup@amdbkp-01:~$ cp -rp production-S3 production-vtape amandabackup@amdbkp-01:~$ cd production-vtape/ amandabackup@amdbkp-01:~$ rm -rf tapelist* disklist s3-statefile curinfo index/* log/*
Define tape type VTAPE and make the size the same as S3 bucket, because we are going to do RAIT with S3 later.
tapetype VTAPE
define tapetype VTAPE {
comment "NFS vTape share"
length 30 gigabytes # Bucket size 30GB
}
Define tpchanger
tpchanger "chg-disk:/vtapes/amanda.vtapes" changerfile "vtapes-statefile" # Amanda will create this file
Create the directory structure and label the first 9 tapes
amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done;
amdevcheck failed until the first backup run. Must be a bug.
amandabackup@amdbkp-01:~$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1 MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory DEVICE_ERROR
Test backup
Check which tape is next in the backup cycle
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amadmin production-vtape tape next
Start the backup (note: we had copied the configuration from S3 configuration directory, so it's already has disklist file with /etc in there) and check its status:
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdump production-vtape &
[1] 10546
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amstatus production-vtape
Using /etc/amanda/production-vtape/log/amdump.1
From Mon Jul 7 12:26:13 IST 2014
amdbkp-01:/etc 0 3361k finished (12:26:14)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 3350k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 3361k 3350k (100.33%) (100.33%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 1 3361k 3350k (100.33%) (100.33%)
tape 1 : 1 3360k 3360k ( 0.01%) PRODUCTION-001 (1 chunks)
4 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 600
holding space : 97673216k (100.00%)
chunker0 busy : 0:00:00 ( 10.25%)
dumper0 busy : 0:00:00 ( 9.59%)
taper busy : 0:00:00 ( 8.04%)
0 dumpers busy : 0:00:01 ( 90.41%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:00 ( 9.59%)
[1]+ Done amdump production-vtape
Now when you do amdevcheck on the slot1 it fails, but succeeds on vtape directory above, because that's where the data file is. Strange!
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1 MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory DEVICE_ERROR amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes SUCCESS
Test restore
Prepare destination directory where we restore into for the test:
root@amdbkp-01:~# mkdir /tmp/1 && cd /tmp
Run the test restore
root@amdbkp-01:/tmp# amrecover production-vtape AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-07) 200 Working date set to 2014-07-07. 200 Config set to production-vtape. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> lcd /tmp/1 amrecover> setdisk /etc 200 Disk set to /etc. amrecover> add inetd.conf Added file /inetd.conf amrecover> extract Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost. The following tapes are needed: PRODUCTION-001 Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost. Load tape PRODUCTION-001 now Continue [?/Y/n/s/d]? Restoring files into directory /tmp/1 All existing files in /tmp/1 can be deleted Continue [?/Y/n]? ./inetd.conf amrecover> quit 200 Good bye.
And now check the file we just restored:
root@amdbkp-01:/tmp# vi 1/inetd.conf
Test encryption
Locate the vTape file. This is 00001.amdbkp-01._etc.0 in our case:
root@amdbkp-01:/tmp# ls /vtapes/amanda.vtapes/slot1/ 00000.PRODUCTION-001 00001.amdbkp-01._etc.0 root@amdbkp-01:/tmp# ls -l /vtapes/amanda.vtapes/slot1/ total 3432 -rw-rw----+ 1 amandabackup disk 32768 Jul 7 12:26 00000.PRODUCTION-001 -rw-rw----+ 1 amandabackup disk 3473520 Jul 7 12:26 00001.amdbkp-01._etc.0
Use dd to inspect the header of the vTape (note: we need only the header hence bs=720)
root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=720 count=1
AMANDA: SPLIT_FILE 20140707122613 amdbkp-01 /etc part 1/-1 lev 0 comp N program /bin/tar crypt enc client_encrypt /usr/sbin/amcrypt-ossl client_decrypt_option -d
ORIGSIZE=3360
DLE=<<ENDDLE
<dle>
<program>GNUTAR</program>
<disk>/etc</disk>
<level>0</level>
<auth>BSDTCP</auth>
<encrypt>CUSTOM<custom-encrypt-program>/usr/sbin/amcrypt-ossl</custom-encrypt-program>
<decrypt-option>-d</decrypt-option>
</encrypt>
<record>YES</record>
<index>YES</index>
<datapath>AMANDA</datapath>
<exclude>
<list>.exclude.lst</list>
</exclude>
</dle>
ENDDLE
To restore, position tape at start of file and run:
dd if=<tape> bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -xpGf - ...
1+0 records in
1+0 records out
720 bytes (720 B) copied, 0.000256799 s, 2.8 MB/s
Test the vTape by extracting the list of files inside:
root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -tvf - drwxr-xr-x root/root 1920 2014-07-07 10:50 ./ drwxr-xr-x root/root 6 2014-04-24 15:25 ./X11/ drwxr-xr-x root/root 1 2012-07-03 07:21 ./X11/xkb/ drwxr-xr-x root/root 22 2014-04-24 15:32 ./acpi/ drwxr-xr-x root/root 11 2014-04-24 15:32 ./acpi/events/ drwxr-xr-x root/root 952 2014-06-11 06:36 ./alternatives/ drwxrwx--- amandabackup/disk 71 2014-07-07 11:04 ./amanda/ <… more output skipped…>
Doing the same without decryption (/usr/sbin/amcrypt-ossl -d) will result in failure:
root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /bin/tar -tvf - /bin/tar: This does not look like a tar archive /bin/tar: Skipping to next header 105+1 records in 105+1 records out /bin/tar: 3440752 bytes (3.4 MB) copiedExiting with failure status due to previous errors , 0.0067797 s, 508 MB/s
RAIT (NAS + S3)
Here is the part of Amnda configuration file amanda.conf, which defines the RAIT.
- WARNING* tapetype should not be inside each changer definition! I took S3 tapetype as the one here, but it can be called whatever.
tpchanger "chg-rait:{nas,s3}"
tapetype S3
define changer nas {
# vTapes
tpchanger "chg-disk:/vtapes/amanda.vtapes"
changerfile "vtapes-statefile" # Amanda will create this file
}
define changer s3 {
# amazonaws S3
device_property "S3_BUCKET_LOCATION" "eu-west-1" # Your S3 bucket location
device_property "S3_ACCESS_KEY" "XXX" # Your S3 Access Key
device_property "S3_SECRET_KEY" "XXX" # Your S3 Secret Key
device_property "S3_SSL" "YES" # Curl needs to have S3 Certification Authority (Verisign today)
# in its CA list. If connection fails, try setting this no NO
tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09}" # Number of tapes in your "tapecycle"
changerfile "s3-statefile" # Amanda will create this file
}
define tapetype S3 {
comment "S3 Bucket"
length 30 gigabytes # Bucket size 30GB
blocksize 1 megabytes
}
tapelist we take from S3 configuration, because RAIT driver won't create tapes for us in S3.
Took the list from S3 configuration directory. Mind the blocksize parameter, it's set in amanda.conf file.
0 PRODUCTION-009 reuse BLOCKSIZE:10240 0 PRODUCTION-008 reuse BLOCKSIZE:10240 0 PRODUCTION-007 reuse BLOCKSIZE:10240 0 PRODUCTION-006 reuse BLOCKSIZE:10240 0 PRODUCTION-005 reuse BLOCKSIZE:10240 0 PRODUCTION-004 reuse BLOCKSIZE:10240 0 PRODUCTION-003 reuse BLOCKSIZE:10240 0 PRODUCTION-002 reuse BLOCKSIZE:10240 0 PRODUCTION-001 reuse BLOCKSIZE:10240
Creating vTapes
Because we had tested vTape and S3 configurations by now, we have tapes in the tow locations having different timestamps and data in it, so RAIT check will be failing. To fix this we need to clean up the both tape locations and re-create tapes.
Removing and re-creating vTapes
amandabackup@amdbkp-01:~$ rm -rf /vtapes/amanda.vtapes/*
amandabackup@amdbkp-01:~$ cd /etc/amanda/production-vtape
amandabackup@amdbkp-01:~$ rm -rf tapelist* vtapes-statefile curinfo index/* log/*
amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done;
Removing and re-creating vTapes in S3
amandabackup@amdbkp-01:~$ cd /etc/amanda/production-S3 amandabackup@amdbkp-01:~$ rm -rf tapelist* s3-statefile curinfo index/* log/*
Login to Amazon S3 management console and remove all tape files from infra/Amanda/production-S3 bucket, then re-create the tapes from the backup server:
amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done;
Copy production-vtape/vtapes-statefile and production-S3/s3-statefile to /etc/amanda/production-S3-vtape-RAIT
amandabackup@amdbkp-01:~$ cat s3-statefile
$STATE = {
'current_slot_by_config' => {
'production-S3-vtape-RAIT' => 's3:infra/Amanda/production-S3/slot-04'
},
'slots' => {
's3:infra/Amanda/production-S3/slot-09' => {
'label' => 'PRODUCTION-009',
'device_status' => '0',
'f_type' => '1',
'state' => 1,
'device_error' => undef
},
's3:infra/Amanda/production-S3/slot-03' => {
'label' => 'PRODUCTION-003',
'device_status' => '0',
'f_type' => '1',
'state' => 1,
'device_error' => undef
},
's3:infra/Amanda/production-S3/slot-04' => {
'device_status' => '0',
'label' => 'PRODUCTION-004',
'f_type' => '1',
'device_error' => undef,
'state' => 1
},
's3:infra/Amanda/production-S3/slot-07' => {
'label' => 'PRODUCTION-007',
'device_status' => '0',
'f_type' => '1',
'state' => 1,
'device_error' => undef
},
's3:infra/Amanda/production-S3/slot-01' => {
'label' => 'PRODUCTION-001',
'device_status' => '0',
'f_type' => '1',
'state' => 1,
'device_error' => undef
},
's3:infra/Amanda/production-S3/slot-06' => {
'device_status' => '0',
'label' => 'PRODUCTION-006',
'f_type' => '1',
'device_error' => undef,
'state' => 1
},
's3:infra/Amanda/production-S3/slot-08' => {
'device_status' => '0',
'label' => 'PRODUCTION-008',
'f_type' => '1',
'device_error' => undef,
'state' => 1
},
's3:infra/Amanda/production-S3/slot-05' => {
'label' => 'PRODUCTION-005',
'device_status' => '0',
'f_type' => '1',
'state' => 1,
'device_error' => undef
},
's3:infra/Amanda/production-S3/slot-02' => {
'device_status' => '0',
'label' => 'PRODUCTION-002',
'f_type' => '1',
'device_error' => undef,
'state' => 1
}
}
};
amandabackup@amdbkp-01:~$ cat vtapes-statefile
$STATE = {
'drives' => {
'/vtapes/amanda.vtapes/drive0' => {}
}
};
If we need more tapes in the RAIT, this article describes how to add more tapes. This requires fiddling with configuration file.
Test backup
Run test backup and check its status:
amandabackup@amdbkp-01:~$ amdump production-S3-vtape-RAIT &
[1] 11922
amandabackup@amdbkp-01:~$ amstatus production-S3-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1
From Mon Jul 7 19:55:30 IST 2014
amdbkp-01:/etc 0 3421k finished (19:55:31)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 3420k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 3421k 3420k (100.03%) (100.03%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 1 3421k 3420k (100.03%) (100.03%)
tape 1 : 1 3420k 3420k ( 0.01%) PRODUCTION-002 (1 chunks)
4 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 600
holding space : 97673216k (100.00%)
chunker0 busy : 0:00:00 ( 11.74%)
dumper0 busy : 0:00:00 ( 10.84%)
taper busy : 0:00:00 ( 54.68%)
0 dumpers busy : 0:00:01 ( 89.16%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:00 ( 10.84%)
Now check that both locations were used. Check S3 and NAS and see that tape 02 was used.
S3:
File:RAIT tape 02 on S3 used.png
NAS:
File:RAIT tape 02 on NAS used.png
Test restore
Prepare test restore destination directory and restore a file into it:
root@amdbkp-01:~# mkdir /tmp/1
root@amdbkp-01:~# cd /tmp/1
root@amdbkp-01:/tmp/1# amrecover production-S3-vtape-RAIT
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-07)
200 Working date set to 2014-07-07.
200 Config set to production-S3-vtape-RAIT.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> lcd /tmp/1
amrecover> add resolv.conf
Added file /resolv.conf
amrecover> extract
Extracting files using tape drive chg-rait:{nas,s3} on host localhost.
The following tapes are needed: PRODUCTION-002
Extracting files using tape drive chg-rait:{nas,s3} on host localhost.
Load tape PRODUCTION-002 now
Continue [?/Y/n/s/d]?
Restoring files into directory /tmp/1
All existing files in /tmp/1 can be deleted
Continue [?/Y/n]?
./resolv.conf
amrecover> quit
200 Good bye.
Check what we have restored:
root@amdbkp-01:/tmp/1# ls -l total 0 lrwxrwxrwx 1 root root 29 Apr 24 15:25 resolv.conf -> ../run/resolvconf/resolv.conf
Test encryption
It's the same process as what we did for S3 and NAS respectively above. Fetch the vTape file from S3 and use dd to inspect it and you can do the same with NAS based vTape file.
RAIT (NAS Datacenter + NAS off-site)
We will create a RAIT over two NAS appliances:
- Office comms room office-nas-01
- Datacenter dc-nas-01
From the previous configuration (NAS+S3) we already have vTapes share mounted off the NAS in Datacenter.
root@amdbkp-01:~# df -k /vtapes Filesystem 1K-blocks Used Available Use% Mounted on dc-nas-01:/test 29056019968 9221493632 19834002048 32% /vtapes
Now we mount NFS share off office based (off-site) NAS:
root@amdbkp-01:~# mkdir /vtapes-offsite root@amdbkp-01:~# mount office-nas-01:/AMANDA /vtapes-offsite root@amdbkp-01:~# df -k | grep vtapes Filesystem 1K-blocks Used Available Use% Mounted on dc-nas-01:/test 29056019968 9222531968 19832963712 32% /vtapes office-nas-01:/AMANDA 5763950016 4348221728 1415204000 76% /vtapes-offsite
Added entries to stab
root@amdbkp-01:~# tail -2 /etc/fstab dc-nas-01:/test /vtapes nfs defaults 2 2 office-nas-01:/AMANDA /vtapes-offsite nfs defaults 2 2
Fixing ownership and permissions on the mounted share:
root@amdbkp-01:~# chown -R amandabackup:disk /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxrwx 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxrwx 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite root@amdbkp-01:~# chmod o-w /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxr-x 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite
Create new Amanda configuration. S3+NAS RAIT is the basis for this one.
root@amdbkp-01:~# su - amandabackup amandabackup@amdbkp-01:~$ cd /etc/amanda amandabackup@amdbkp-01:/etc/amanda$ ls amanda-client.conf production-S3 production-S3-vtape-RAIT production-tape production-vtape amandabackup@amdbkp-01:/etc/amanda$ cp -rp production-S3-vtape-RAIT production-vtape-RAIT amandabackup@amdbkp-01:/etc/amanda$ cd production-vtape-RAIT/ amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l total 21 -rw-rw-r-- 1 amandabackup disk 7732 Jul 7 20:17 amanda.conf -rw------- 1 amandabackup disk 135 Jul 7 17:01 changer drwxr-xr-x 3 amandabackup disk 1024 Jul 7 19:48 curinfo -rw-r--r-- 1 amandabackup disk 45 Jul 7 12:11 disklist drwxrwxr-x 3 amandabackup disk 1024 Jul 7 19:48 index drwxrwxr-x 3 amandabackup disk 1024 Jul 7 20:57 log -rw------- 1 amandabackup disk 5784 Jul 7 22:44 s3-statefile -rw------- 1 amandabackup disk 472 Jul 7 20:57 tapelist -rw------- 1 amandabackup disk 0 Jul 7 19:48 tapelist.lock -rw------- 1 amandabackup disk 135 Jul 7 22:44 vtapes-statefile amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ mkdir /vtapes/amanda.vtapes-RAIT amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -ld /vtapes/amanda.vtapes-RAIT drwxrwx---+ 2 amandabackup disk 4096 Jul 16 16:42 /vtapes/amanda.vtapes-RAIT
Clean up configuration directory off S3+NAS files
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ rm -rf tapelist* vtapes-statefile curinfo index/* log/* s3-statefile changer amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l total 11 -rw-rw-r-- 1 amandabackup disk 7732 Jul 7 20:17 amanda.conf -rw-r--r-- 1 amandabackup disk 45 Jul 7 12:11 disklist drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 index drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 log
Edit Amanda file. We define the two changers and the RAIT changer based on the two:
tpchanger "chg-rait:{nas-tc,nas-gqh}"
tapetype VTAPE
define changer nas-tc {
tpchanger "chg-disk:/vtapes/amanda.vtapes-RAIT"
changerfile "vtapes-statefile-tc" # Amanda will create this file
}
define changer nas-gqh {
tpchanger "chg-disk:/vtapes-offsite"
changerfile "vtapes-statefile-gqh" # Amanda will create this file
}
define tapetype VTAPE {
comment "NFS vTape share"
length 30 gigabytes # Bucket size 30GB
}
Creating vTapes
Create vTapes and label it. The good thing about two NAS location is that you can do this in one go for both (not like with S3 and NAS, where you had to create vTapes for each configuration separately)
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes-RAIT/slot${i} /vtapes-offsite/slot${i} && amlabel production-vtape-RAIT PRODUCTION-00$i slot $i; done;
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-001'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-002'...
Checking label...
Success!
<...skipped similar output for the rest of the vTapes...>
Check the results in both NAS units.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/ /vtapes-offsite/ /vtapes-offsite/slot1 /vtapes-offsite/slot1/00000.PRODUCTION-001 /vtapes-offsite/slot8 /vtapes-offsite/slot8/00000.PRODUCTION-008 /vtapes-offsite/slot4 /vtapes-offsite/slot4/00000.PRODUCTION-004 /vtapes-offsite/slot3 /vtapes-offsite/slot3/00000.PRODUCTION-003 /vtapes-offsite/slot6 /vtapes-offsite/slot6/00000.PRODUCTION-006 /vtapes-offsite/slot2 /vtapes-offsite/slot2/00000.PRODUCTION-002 /vtapes-offsite/slot7 /vtapes-offsite/slot7/00000.PRODUCTION-007 /vtapes-offsite/slot5 /vtapes-offsite/slot5/00000.PRODUCTION-005 /vtapes-offsite/slot9 /vtapes-offsite/slot9/00000.PRODUCTION-009 amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/slot2 /vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002 /vtapes/amanda.vtapes-RAIT/slot5 /vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005 /vtapes/amanda.vtapes-RAIT/slot1 /vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001 /vtapes/amanda.vtapes-RAIT/slot4 /vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004 /vtapes/amanda.vtapes-RAIT/slot8 /vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008 /vtapes/amanda.vtapes-RAIT/slot7 /vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007 /vtapes/amanda.vtapes-RAIT/slot9 /vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009 /vtapes/amanda.vtapes-RAIT/slot6 /vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006 /vtapes/amanda.vtapes-RAIT/slot3 /vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003
Checking configuration. Ignore the error about /etc/.exclude.lst missing, we don't have it yet (and may not even need it)
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB
slot {1,1}: volume 'PRODUCTION-001'
Will write to volume 'PRODUCTION-001' in slot {1,1}.
NOTE: skipping tape-writable test
Server check took 0.196 seconds
Amanda Backup Client Hosts Check
--------------------------------
ERROR: amdbkp-01: [Can't open exclude file /etc/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 2.066 seconds. 1 problem found.
(brought to you by Amanda 3.3.5)
Test backup
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amdump production-vtape-RAIT &
[1] 11946
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1
From Wed Jul 16 16:54:46 IST 2014
amdbkp-01:/etc 0 3561k finished (16:54:48)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 3560k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 3561k 3560k (100.03%) (100.03%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 1 3561k 3560k (100.03%) (100.03%)
tape 1 : 1 3560k 3560k ( 0.01%) PRODUCTION-001 (1 chunks)
4 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 600
holding space : 97673216k (100.00%)
chunker0 busy : 0:00:00 ( 23.85%)
dumper0 busy : 0:00:00 ( 23.28%)
taper busy : 0:00:00 ( 12.77%)
0 dumpers busy : 0:00:01 ( 76.63%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:00 ( 23.28%)
[1]+ Done amdump production-vtape-RAIT
Check the result (new tape file in slot1) in both NAS.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/ /vtapes-offsite/ /vtapes-offsite/slot1 /vtapes-offsite/slot1/00001.amdbkp-01._etc.0 /vtapes-offsite/slot1/00000.PRODUCTION-001 /vtapes-offsite/data /vtapes-offsite/slot8 /vtapes-offsite/slot8/00000.PRODUCTION-008 /vtapes-offsite/slot4 /vtapes-offsite/slot4/00000.PRODUCTION-004 /vtapes-offsite/slot3 /vtapes-offsite/slot3/00000.PRODUCTION-003 /vtapes-offsite/slot6 /vtapes-offsite/slot6/00000.PRODUCTION-006 /vtapes-offsite/slot2 /vtapes-offsite/slot2/00000.PRODUCTION-002 /vtapes-offsite/slot7 /vtapes-offsite/slot7/00000.PRODUCTION-007 /vtapes-offsite/slot5 /vtapes-offsite/slot5/00000.PRODUCTION-005 /vtapes-offsite/slot9 /vtapes-offsite/slot9/00000.PRODUCTION-009 amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/slot2 /vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002 /vtapes/amanda.vtapes-RAIT/slot5 /vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005 /vtapes/amanda.vtapes-RAIT/slot1 /vtapes/amanda.vtapes-RAIT/slot1/00001.amdbkp-01._etc.0 /vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001 /vtapes/amanda.vtapes-RAIT/slot4 /vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004 /vtapes/amanda.vtapes-RAIT/slot8 /vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008 /vtapes/amanda.vtapes-RAIT/slot7 /vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007 /vtapes/amanda.vtapes-RAIT/slot9 /vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009 /vtapes/amanda.vtapes-RAIT/slot6 /vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006 /vtapes/amanda.vtapes-RAIT/data /vtapes/amanda.vtapes-RAIT/slot3 /vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003
Let's get some bigger files to test backup speed. There are some big TSM files in nas-02:/test, let's grub those.
root@amdbkp-01:~# mount nas-02:/test /mnt
We will need a separate filesystem for the test. Check free space left in the VG.
root@amdbkp-01:~# vgdisplay --- Volume group --- VG Name amdbkp-01-vg System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 12 VG Access read/write VG Status resizable MAX LV 0 Cur LV 7 Open LV 7 Max PV 0 Cur PV 2 Act PV 2 VG Size 272.98 GiB PE Size 4.00 MiB Total PE 69883 Alloc PE / Size 36243 / 141.57 GiB Free PE / Size 33640 / 131.41 GiB VG UUID ENArtj-VebI-x5WG-Lpc0-zN5X-cLpL-FRsT9W
Create new LV and filesystem.
root@amdbkp-01:~# lvcreate -n testlv -L10G amdbkp-01-vg
Logical volume "testlv" created
root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/testlv
root@amdbkp-01:~# mkdir /test
root@amdbkp-01:~# mount /dev/amdbkp-01-vg/testlv /test
root@amdbkp-01:~# chmod 0755 /test
root@amdbkp-01:~# cp /mnt/TSM71/{TSM_7.1_WIN_*,TSM_7.1_LIN86_AGT_ML.bin} /test/
root@amdbkp-01:~# df -k /tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/amdbkp--01--vg-testlv 10190136 4868312 4797536 51% /test
Add /test to disklist.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ cat /etc/amanda/production-vtape-RAIT/disklist #amdbkp-01 /etc tar-client-encrypt amdbkp-01 /test tar-client-encrypt
Check configuration for errors.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB
slot {1,1}: volume 'PRODUCTION-001' is still active and cannot be overwritten
slot {2,2}: volume 'PRODUCTION-002'
Will write to volume 'PRODUCTION-002' in slot {2,2}.
NOTE: skipping tape-writable test
NOTE: info dir /etc/amanda/production-S3-vtape-RAIT/curinfo/amdbkp-01/_test does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/production-S3-vtape-RAIT/index/amdbkp-01/_test does not exist
NOTE: it will be created on the next run.
Server check took 0.220 seconds
Amanda Backup Client Hosts Check
--------------------------------
ERROR: amdbkp-01: [Can't open exclude file /test/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 1.095 seconds. 1 problem found.
(brought to you by Amanda 3.3.5)
Run dump of /test. I thought it was going to take some time and I'd have to walk away and check the result later, so I used 'nohup'.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ nohup amdump production-vtape-RAIT &
Checking the status of the dump and it's impressively fast. Mind you, it's the percentage completed to holding disk (!) not the vTapes yet.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014
amdbkp-01:/test 0 4838040k dumping 1727328k ( 35.70%) (17:13:36)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 4838040k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 1 1727328k 4838040k ( 35.70%) ( 35.70%)
dumped : 0 0k 0k ( 0.00%) ( 0.00%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 0 0k 0k ( 0.00%) ( 0.00%)
3 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 0
holding space : 92834976k ( 95.05%)
dumper0 busy : 0:00:00 ( 0.00%)
0 dumpers busy : 0:00:01 (100.00%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:00 ( 0.00%)
And now it's writing to vTapes.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014
amdbkp-01:/test 0 4838041k dump done, writing to tape (1241440k done (25.66%)) (17:14:17)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 4838040k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 4838041k 4838040k (100.00%) (100.00%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 1 4838041k 4838040k (100.00%) (100.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 0 0k 0k ( 0.00%) ( 0.00%)
tape 1 : 0 0k 0k ( 0.00%) PRODUCTION-002
4 dumpers idle : 0
taper status: Writing amdbkp-01:/test
taper qlen: 0
network free kps: 600
holding space : 92835015k ( 95.05%)
chunker0 busy : 0:00:41 ( 97.37%)
dumper0 busy : 0:00:41 ( 97.35%)
0 dumpers busy : 0:00:01 ( 2.65%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:41 ( 97.35%) 0: 0:00:41 (100.00%)
Done in 41 seconds (!). What was strange though is that it seemed to have being writing to NAS still even after amstatus reported 100% completion. Need to test it again. When I run 'df -k' right after amstatus reported backup complete, it stayed there for about a minute before I saw df output. And at the same time observing inbound traffic to the office QNAP (via web UI System status -> Bandwidth usage), it was showing ~9MB/s inboud traffic even after amstatus reported 100% completion.
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014
amdbkp-01:/test 0 4838041k dump done, writing to tape (4838048k done (100.00%)) (17:14:17)
SUMMARY part real estimated
size size
partition : 1
estimated : 1 4838040k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 0 0k ( 0.00%)
dumping to tape : 0 0k ( 0.00%)
dumping : 0 0k 0k ( 0.00%) ( 0.00%)
dumped : 1 4838041k 4838040k (100.00%) (100.00%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 1 4838041k 4838040k (100.00%) (100.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 0 0k 0k ( 0.00%) ( 0.00%)
tape 1 : 0 0k 0k ( 0.00%) PRODUCTION-002
4 dumpers idle : 0
taper status: Writing amdbkp-01:/test
taper qlen: 0
network free kps: 600
holding space : 92835015k ( 95.05%)
chunker0 busy : 0:00:41 ( 97.37%)
dumper0 busy : 0:00:41 ( 97.35%)
0 dumpers busy : 0:00:01 ( 2.65%) 0: 0:00:01 (100.00%)
1 dumper busy : 0:00:41 ( 97.35%) 0: 0:00:41 (100.00%)
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -la /vtapes-offsite/slot2/
total 4838132
drwxr-xr-x 2 amandabackup disk 4096 Jul 16 17:14 .
drwxrwxr-x 12 amandabackup disk 4096 Jul 16 17:13 ..
-rw------- 1 amandabackup disk 32768 Jul 16 17:14 00000.PRODUCTION-002
-rw------- 1 amandabackup disk 4954193920 Jul 16 17:22 00001.amdbkp-01._test.0
I wasn's sure if amanda process was gone and not writing anything at that stage, so I checked it:
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ps -ef | grep am root 11046 897 0 16:31 ? 00:00:00 sshd: alex [priv] alex 11301 11046 0 16:31 ? 00:00:00 sshd: alex@pts/1 alex 11302 11301 0 16:31 pts/1 00:00:00 -bash 63998 12375 11405 0 17:12 pts/1 00:00:00 su - amandabackup 63998 12598 12376 0 17:22 pts/1 00:00:00 grep am [1]+ Done nohup amdump production-vtape-RAIT
Test restore
Test encryption
Amanda clients
Firewall policies
You need the following policies in the firewall to allow Amanda server to talk to client and client to talk back to server. Below is an example for U3 pre-prod.
File:Amanda firewall rules IN.png
File:Amanda firewall rules OUT.png
If you want to setup iptables for the same, please, follow this article.
Linux
Linux (CentOS)
Client installation
- NOTE* Linux server infra-mon-02 was used for this example deployment.
Download the package from Amanda site.
[root@infra-mon-02 ]# cd /tmp && wget -c -t0 http://www.zmanda.com/downloads/community/Amanda/3.3.5/Redhat_Enterprise_6.0/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm
Install pre-requisite packages:
[root@infra-mon-02 ]# yum install xinetd perl-JSON
Install Amanda backup client
[root@infra-mon-02 ]# rpm -Uhv /tmp/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm
Add Amanda server line to client's /var/lib/amanda/.amandahosts file
localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump
Start xinetd and confirm that it's listening on Amanda client port 10080:
[root@infra-mon-02 ]# /etc/init.d/xinetd start [root@infra-mon-02 ]# netstat -anp | grep 1008 tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 2647/xinetd
Linux (Ubuntu)
Client installation
- NOTE* Linux server nas2ftp-01 was used for this example deployment
Download the package from Amanda site.
# cd /tmp && wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb
Install pre-requisite packages gettext and xinetd before you attempt to install Amanda client
nas2ftp-01 # apt-get update nas2ftp-01 # apt-get install get text xinetd
Now, install Amanda client
nas2ftp-01 # dpkg -i /tmp/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb
Add Amanda server line to client's /var/lib/amanda/.amandahosts file
localhost root amindexd amidxtaped localhost.localdomain root amindexd amidxtaped localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump
If you have iptables active on the client, add the below line to the rule set (usually, it's in /etc/iptables.active.rules, if the client was provisioned from Linux VM template)
# # allow Amanda in -A INPUT -s 192.168.0.23 -j ACCEPT
Re-load iptables rules
nas2ftp-01 # iptables-restore < /etc/iptables.active.rules
Backup configuration on backup server
Add what you want to backup to disklist file on Amanda server (below, I added /var/spool/cron/crontabs directory from nas2ftp-01 to be backup)
nas2ftp-01 /var/spool/cron/crontabs/root tar-client-encrypt
Run check to make sure that everything is fine from Amanda server prospective (you can ignore warning about absence of .exclude.lst file, if you don't use it to exclude something from backup):
amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amcheck prod-1week-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 188 GB disk space available, using 188 GB
slot {8,8}: volume 'PRODUCTION-1WEEK-008'
Will write to volume 'PRODUCTION-1WEEK-008' in slot {8,8}.
NOTE: skipping tape-writable test
NOTE: host info dir /etc/amanda/prod-1week-vtape-RAIT/curinfo/nas2ftp-01 does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/prod-1week-vtape-RAIT/index/nas2ftp-01 does not exist
NOTE: it will be created on the next run.
Server check took 0.192 seconds
Amanda Backup Client Hosts Check
--------------------------------
ERROR: nas2ftp-01: [Can't open exclude file /var/spool/cron/crontabs/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 2.099 seconds. 1 problem found.
(brought to you by Amanda 3.3.5)
Now, we can run a test backup
amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amdump prod-1week-vtape-RAIT nas2ftp-01 &
[1] 17270
amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amstatus prod-1week-vtape-RAIT
Using /etc/amanda/prod-1week-vtape-RAIT/log/amdump
From Wed Oct 8 14:57:18 IST 2014
nas2ftp-01:/var/spool/cron/crontabs 0 0g estimate done
SUMMARY part real estimated
size size
partition : 1
estimated : 1 0g
flush : 0 0g
failed : 0 0g ( 0.00%)
wait for dumping: 0 0g ( 0.00%)
dumping to tape : 0 0g ( 0.00%)
dumping : 0 0g 0g ( 0.00%) ( 0.00%)
dumped : 0 0g 0g ( 0.00%) ( 0.00%)
wait for writing: 0 0g 0g ( 0.00%) ( 0.00%)
wait to flush : 0 0g 0g (100.00%) ( 0.00%)
writing to tape : 0 0g 0g ( 0.00%) ( 0.00%)
failed to tape : 0 0g 0g ( 0.00%) ( 0.00%)
taped : 0 0g 0g ( 0.00%) ( 0.00%)
4 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 80000
holding space : 188g (100.00%)
0 dumpers busy : 0:00:00 (100.00%)
Shortly after you will have email report to say that the backup was good:
Hostname: amdbkp-01
Org : production. 1 week cycle.
Config : prod-1week-vtape-RAIT
Date : October 8, 2014
These dumps were to tape PRODUCTION-1WEEK-008.
The next 3 tapes Amanda expects to use are: 3 new tapes.
The next 3 new tapes already labelled are: PRODUCTION-1WEEK-009, PRODUCTION-1WEEK-010, PRODUCTION-1WEEK-011
STATISTICS:
Total Full Incr. Level:#
-------- -------- -------- --------
Estimate Time (hrs:min) 0:00
Run Time (hrs:min) 0:00
Dump Time (hrs:min) 0:00 0:00 0:00
Output Size (meg) 0.0 0.0 0.0
Original Size (meg) 0.0 0.0 0.0
Avg Compressed Size (%) 110.0 110.0 --
DLEs Dumped 1 1 0
Avg Dump Rate (k/s) 89.4 89.4 --
Tape Time (hrs:min) 0:00 0:00 0:00
Tape Size (meg) 0.0 0.0 0.0
Tape Used (%) 0.0 0.0 0.0
DLEs Taped 1 1 0
Parts Taped 1 1 0
Avg Tp Write Rate (k/s) 100.0 100.0 --
USAGE BY TAPE:
Label Time Size % DLEs Parts
PRODUCTION-1WEEK-008 0:00 0G 0.0 1 1
NOTES:
planner: Adding new disk nas2ftp-01:/var/spool/cron/crontabs.
taper: Slot {8,8} with label PRODUCTION-1WEEK-008 is usable
taper: tape PRODUCTION-1WEEK-008 kb 10 fm 1 [OK]
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-GB OUT-GB COMP% MMM:SS KB/s MMM:SS KB/s
------------------------------------------- ---------------------- -------------- -------------
nas2ftp-01 /var/spool/cron/crontabs 0 0 0 -- 0:00 89.4 0:00 100.0
(brought to you by Amanda version 3.3.5)
Windows client
Client installation
Ref:
There is amandabackup user account in PRODUCTION Windows domain created for Amanda client.
Install Amanda client binaries:
File:Windows client installer Amanda.PNG
Supply amandabackup user password when prompted
File:Windows client Amanda password prompt.PNG
Enter Amanda server name when prompted:
File:Windows client Amanda server prompt.PNG
Check the amandabackup user is a member of local Administrators group on the Windows server:
File:Windows client Amanda server local admins.PNG
Backup configuration on backup server
On Amanda server register the new client in the configuration that it meant to be in (prod-selligent-1week-vtape-RAIT configuration in the example below):
amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amaddclient --config prod-selligent-1week-vtape-RAIT --client selrep-01.production --diskdev "E:/SSRS/SelligentReport" --dumptype zwc-compress Logging to /var/log/amanda/amaddclient.20141009171134.debug /etc/amanda/prod-selligent-1week-vtape-RAIT/disklist updated updating /var/lib/amanda/.amandahosts on amdbkp-01.production Attempting to update /var/lib/amanda/.amandahosts on selrep-01.production ssh: connect to host selrep-01.production port 22: Connection refused WARNING: scp from selrep-01.production not successful. Check selrep-01.production :/var/lib/amanda/.amandahosts file. If entry 'amdbkp-01.production amandabackup' is not present, append the entry to the file manually. Creating amanda-client.conf for selrep-01.production Creating /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production ssh: connect to host selrep-01.production port 22: Connection refused WARNING: Cannot create /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production Please copy /var/lib/amanda/amanda-client.conf-selrep-01.production to selrep-01.production manually File /var/lib/amanda/example/xinetd.amandaclient contains the latest Amanda client daemon configuration. Please merge it to /etc/xinetd.d/amandaclient.
Run client check to make sure everything is working fine:
amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amcheck -c prod-selligent-1week-vtape-RAIT Amanda Backup Client Hosts Check -------------------------------- Client check: 1 host checked in 0.236 seconds. 0 problems found. (brought to you by Amanda 3.3.5)
Backup profile creation
Now we want to add new backup profile for new application.
- NOTE* Below I took BI as an example of the new profile
You will find new profile creation script in /etc/amanda directory - new_profile.sh. The script will:
- create Amanda profile directory
- create two configuration files - with and without RAIT defined
- create holding disk area for the profile
- create specified number of vTapes on both NAS units - onsite and offsite
Source of the script:
#!/bin/sh
#
# The script creates new profile for Amanda backup
# based on template file /var/lib/amanda/template.d/amanda--rait.conf
#
# NOTE: you might need to change the following by hand:
# * dumpcycle (default: 1 weeks)
# * runspercycle (default: 7)
# * tapecycle (default: 14 tapes)
# * tapetype (default: VTAPE-100G)
#
PROFILE_NAME=$1
NUM_OF_VTAPES=$2
PROFILE_NAME_LOW=`echo $PROFILE_NAME | awk '{print tolower($1)}'`
PROFILE_NAME_UP=`echo $PROFILE_NAME | awk '{print toupper($1)}'`
TEMPLATE=/var/lib/amanda/template.d/amanda-.conf
TEMPLATE_RAIT=/var/lib/amanda/template.d/amanda--rait.conf
AMANDA_CFG_DIR=/etc/amanda
AMANDA_HOLDING_DISK_DIR=/amanda
AMANDA_PROFILE_DIR=prod-$PROFILE_NAME_LOW-1week-vtape-RAIT
VTAPES_DIR=/vtapes-prod/RAIT/1week
VTAPES_OFFSITE_DIR=/vtapes-prod-offsite/RAIT/1week
WARNING="NOTE: you might need to change the following by hand in $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf:\n
dumpcycle (default: 1 weeks)\n
runspercycle (default: 7)\n
tapecycle (default: 14 tapes)\n
tapetype (default: VTAPE-100G)\n"
DISKLIST_HDR="
#\n
# The below configured based on the requirements I had at that moment\n
#\n"
if [ $# -lt 2 ]
then
echo "\nUsage: `basename $0` <new_profile_name> <number_of_vtapes>\n"
exit 1
fi
if [ -f $TEMPLATE ]
then
if [ -d $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR ]
then
echo "\nERROR: $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR already exists!\n"
exit 1
fi
echo "Creating profile directory $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR \n"
mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR
# configuration without RAIT
sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait
# configuration with RAIT
sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE_RAIT > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait
# until we finished creating vTapes, we need RAIT configuration to be active
ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf
for d in log curinfo index
do
mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/${d}
done
echo $DISKLIST_HDR > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/disklist
HOLDING_DISK_AREA=`amgetconf $AMANDA_PROFILE_DIR holdingdisk:hd1:directory`
echo "Creating holding disk area for the profile - $HOLDING_DISK_AREA\n"
if [ -d $AMANDA_HOLDING_DISK_DIR ]
then
if [ ! -d $HOLDING_DISK_AREA ]
then
mkdir $HOLDING_DISK_AREA
else
echo "Directory $HOLDING_DISK_AREA already exists!"
fi
else
echo "ERROR: Holding disk directory $AMANDA_HOLDING_DISK_DIR does not exist!"
fi
echo "Creating $NUM_OF_VTAPES vTapes for the profile"
for i in `seq -f "%03g" 1 $NUM_OF_VTAPES`
do
SN=`echo ${i}|sed -e 's/0*\([1-9].*\)/\1/g'`
mkdir -p $VTAPES_DIR/$PROFILE_NAME_UP/slot$SN $VTAPES_OFFSITE_DIR/$PROFILE_NAME_UP/slot$SN && amlabel $AMANDA_PROFILE_DIR PRODUCTION-$PROFILE_NAME_UP-1WEEK-$i slot $SN
done
# switching to configuration without RAIT
# should upgrade DC - HQ connection before we can use RAIT for all applications
# for now RAIT is not in use.
rm $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf
ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf
echo "Profile $PROFILE_NAME_UP created!\n\n"
echo $WARNING
fi
Engage the script to create BI profile with 14 vTapes:
amandabackup@amdbkp-01:/etc/amanda$ ./new_profile.sh BI 14 Creating profile directory /etc/amanda/prod-sql-1week-vtape-RAIT Creating holding disk area for the profile - /amanda/prod-sql-1week-vtape-RAIT Directory /amanda/prod-sql-1week-vtape-RAIT already exists! Creating 14 vTapes for the profile Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-001'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-002'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-003'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-004'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-005'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-006'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-007'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-008'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-009'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-010'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-011'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-012'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-013'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-014'... Checking label... Success! Profile BI created! NOTE: you might need to change the following by hand in /etc/amanda/prod-sql-1week-vtape-RAIT/amanda.conf: dumpcycle (default: 1 weeks) runspercycle (default: 7) tapecycle (default: 14 tapes) tapetype (default: VTAPE-100G)
Newly created amanda.conf points to Datacenter NAS only and amanda.conf-rait has RAIT (Datacenter NAS + HQ NAS) defined.
Here is the diff between the two templates :
--- /var/lib/amanda/template.d/amanda-.conf 2014-10-20 16:43:59.794381389 +0100
+++ /var/lib/amanda/template.d/amanda--rait.conf 2014-10-20 16:45:09.847107263 +0100
@@ -16,9 +16,16 @@
bumpmult 4
etimeout 1800
runtapes 3
-tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP"
-changerfile "vtapes-statefile-tc"
+tpchanger "chg-rait:{nas-tc,nas-gqh}"
tapetype VTAPE-100G
+define changer nas-tc {
+ tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP"
+ changerfile "vtapes-statefile-tc"
+}
+define changer nas-gqh {
+ tpchanger "chg-disk:/vtapes-prod-offsite/RAIT/1week/PROFILE_NAME_UP"
+ changerfile "vtapes-statefile-gqh"
+}
maxdumpsize -1
labelstr "^PRODUCTION-PROFILE_NAME_UP-1WEEK-[0-9][0-9]*$"
holdingdisk hd1 {
- WARNING* we should upgrade Datacenter - HQ connection before we can use RAIT for all applications. For now RAIT is not in use.
Add required backup targets to disklist file.
# # The below configured based on the requirements # SQL-01 I:/OBI/catalog zwc-encrypt-compress-server SQL-01 I:/OBI/GlobalCache zwc-encrypt-compress-server SQL-01 I:/OBI/repository zwc-encrypt-compress-server SQL-01 I:/Sources zwc-encrypt-compress-server
Check that configuration is correct and working (ignore "does not exist" messages for info and index):
amandabackup@amdbkp-01:/etc/amanda$ amcheck prod-sql-1week-vtape-RAIT Amanda Tape Server Host Check ----------------------------- Holding disk /amanda/prod-sql-1week-vtape-RAIT: 148107264 KB disk space available, using 148004864 KB slot 1: volume 'PRODUCTION-SQL-1WEEK-001' Will write to volume 'PRODUCTION-SQL-1WEEK-001' in slot 1. NOTE: skipping tape-writable test NOTE: host info dir /etc/amanda/prod-sql-1week-vtape-RAIT/curinfo/SQL-01 does not exist NOTE: it will be created on the next run. NOTE: index dir /etc/amanda/prod-sql-1week-vtape-RAIT/index/SQL-01 does not exist NOTE: it will be created on the next run. Server check took 0.174 seconds Amanda Backup Client Hosts Check -------------------------------- Client check: 1 host checked in 0.301 seconds. 0 problems found. (brought to you by Amanda 3.3.5)
Create distribution list in you directory
File:Amanda BI distribution list.png
Add members to the list.
Run backup and watch results:
amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ nohup /etc/amanda/amdump_and_report.sh prod-sql-1week-vtape-RAIT &
amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ watch -n3 amstatus prod-sql-1week-vtape-RAIT
Every 3.0s: amstatus prod-sql-1week-vtape-RAIT Mon Oct 20 17:07:14 2014
Using /etc/amanda/prod-sql-1week-vtape-RAIT/log/amdump
From Mon Oct 20 17:06:56 IST 2014
SQL-01:"I:/OBI/GlobalCache" 0 1k finished (17:07:04)
SQL-01:"I:/OBI/catalog" 0 6001k dumping 672k ( 11.20%) (17:07:09)
SQL-01:"I:/OBI/repository" 0 2339k finished (17:07:07)
SQL-01:"I:/Sources" 0 357273k waiting for dumping
SUMMARY part real estimated
size size
partition : 4
estimated : 4 364485k
flush : 0 0k
failed : 0 0k ( 0.00%)
wait for dumping: 1 357273k ( 98.02%)
dumping to tape : 0 0k ( 0.00%)
dumping : 1 672k 6001k ( 11.20%) ( 0.18%)
dumped : 2 2340k 1211k (193.23%) ( 0.64%)
wait for writing: 0 0k 0k ( 0.00%) ( 0.00%)
wait to flush : 0 0k 0k (100.00%) ( 0.00%)
writing to tape : 0 0k 0k ( 0.00%) ( 0.00%)
failed to tape : 0 0k 0k ( 0.00%) ( 0.00%)
taped : 2 2340k 1211k (193.23%) ( 0.64%)
tape 1 : 2 2338k 2338k ( 0.00%) PRODUCTION-SQL-1WEEK-001 (2 chunks)
3 dumpers idle : 0
taper status: Idle
taper qlen: 0
network free kps: 78976
holding space : 147998784k (100.00%)
chunker0 busy : 0:00:08 ( 60.97%)
dumper0 busy : 0:00:08 ( 60.80%)
taper busy : 0:00:00 ( 0.35%)
0 dumpers busy : 0:00:05 ( 39.20%) 0: 0:00:03 ( 60.64%)
2: 0:00:02 ( 39.34%)
1 dumper busy : 0:00:08 ( 60.80%) 0: 0:00:05 ( 67.10%)
5: 0:00:02 ( 32.90%)
Since everything is confirmed to be working, you can add that distribution list you created in directory to the "mailto" parameter in amanda.conf* files.
amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ grep mailto amanda.conf* amanda.conf:mailto "backup-reports@corporate eng@corporate" amanda.conf-norait:mailto "backup-reports@corporate eng@corporate" amanda.conf-rait:mailto "backup-reports@corporate eng@corporate"
Logging and monitoring
Kibana/Logstash
We had Kibana and Logstash deployed and you could see logs from every run of Amanda backup in Kibana. See example of wrapper script we had to send Amanda reports in JSON format to see it later in Kibana.
Parsing of JSON output from Amanda requires improvement. We had few issues to resolve there:
- There is a known limitation of Logstash - it can't explode JSON arrays properly. It leads to "summary" part of the output not parsed properly.
- Amounts sent by Amanda is in kilobytes, but Kibana works with bytes and doesn't allow to do mathematical operations with the numbers in web UI. There is an option of doing it in the input filter at logstash level, but it would be preferable not to alter the original data on the way into logstash. After some discussion we thought that the better way was to ship the stats into Graphite and use Dashboard to plot the graphs (it allows calculations in web UI)
Dashboard
We had an idea to export data into Graphite, so we could build dashboards for backups.
Amanda server statistics were available via Amanda backup server dashboard.
