View source

From MyWiki

You do not have permission to edit this page, for the following reasons:

• The action you have requested is limited to users in the group: Administrators.
• This page has been locked to prevent editing.

You can view and copy the source of this page:

==Amanda== ===Preparing the server=== ====Syslog settings==== We need to configure Syslog to send messages out to Logstash through Redis in RELP format. <pre> root@amdbkp-01:~# apt-get update root@amdbkp-01:~# apt-get install python-software-properties root@amdbkp-01:~# add-apt-repository ppa:adiscon/v8-stable You are about to add the following PPA to your system: Contains the latest RSyslog V8-Stable packages and dependencies. More info: https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable Press [ENTER] to continue or ctrl-c to cancel adding it gpg: keyring `/tmp/tmpLRDNaP/secring.gpg' created gpg: keyring `/tmp/tmpLRDNaP/pubring.gpg' created gpg: requesting key 5234BF2B from hkp server keyserver.ubuntu.com gpg: /tmp/tmpLRDNaP/trustdb.gpg: trustdb created gpg: key 5234BF2B: public key "Launchpad PPA for Adiscon" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK </pre> Another <tt>apt-get update</tt> <pre> root@amdbkp-01:~# apt-get update </pre> Install RELP library and rsyslog <pre> root@amdbkp-01:~# apt-get install librelp0 rsyslog rsyslog-relp </pre> Create configuration file to send messages to Redis in RELP format <pre> root@amdbkp-01:~# vi /etc/rsyslog.d/sendsyslog2logstash.conf </pre> Add this line into the file, save and exit. <pre> action(type="omrelp" target="redis-m.production" port="2514") </pre> Add RELP configuration to <tt>/etc/rsyslog.conf</tt> file (add the below lines into Modules section): <pre> #RELP Output module(load="omrelp") </pre> Turn off repeated message reduction, so you have every message instead of those pesky "message repeated N times" lines. <pre> # Filter duplicated messages $RepeatedMsgReduction off </pre> Restart Syslog <pre> root@amdbkp-01:~# service rsyslog restart </pre> ====Sysctl settings==== Some of the estimates from clients may run for some time (tens of minutes) and TCP connection might be killed due to inactivity by Fortigate firewall. Turns out Fortigates have a default setting of five minutes (see [http://www.ispcolohost.com/2014/09/26/keep-losing-your-ssh-session-from-behind-a-fortigate/ the article]). Refer to [http://wiki.zmanda.com/index.php/Mesg_read:_Connection_reset_by_peer Connection reset by peer] article for more details. We need to reduce net.ipv4.tcp_keepalive_time from default 7200 to 180 to survive that. Put the below into <tt>/etc/sysctl.conf</tt>: <pre> # Amanda related, please, do not remove! # Ref http://wiki.zmanda.com/index.php/Mesg_read:_Connection_reset_by_peer net.ipv4.tcp_keepalive_time = 180 </pre> Change it while the server is still running: <pre> root@amdbkp-01:~# echo 180 > /proc/sys/net/ipv4/tcp_keepalive_time </pre> ====Installing the software==== *WARNING* Restore test revealed problem with the Amanda 3.3.0 included by default with Ubuntu 12.04. Ended up installing the latest version of Amanda 3.3.5 downloaded directly [http://www.zmanda.com/download-amanda.php from Amanda]. Thanks to [https://forums.zmanda.com/archive/index.php/t-4824.html this thread] for the hint! First we pull packages from [http://www.zmanda.com/download-amanda.php Amanda site] and remove default Ubuntu 12.04 LTS Amanda packages (if installed). <pre> root@amdbkp-01:~# cd /tmp root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb . root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb . root@amdbkp-01:~# apt-get remove amanda-server amanda-client amanda-common </pre> This packages depend on <b>xinetd</b> and <b>get text</b> been installed. Since we are not using xinetd, we will ignore it, but install gettext =====Inetd way===== If you want to stay with inetd. Mind you, it will create problem with 'apt-get install' later. <pre> root@amdbkp-01:~# apt-get will be throwing this error later root@amdbkp-01:~# apt-get install gettext root@amdbkp-01:~# dpkg -i --ignore-depends=xinetd ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb </pre> Put back the line to inetd.conf. You don't really need this until you want to stick with inetd. <pre> root@amdbkp-01:~# tail -1 /etc/inetd.conf amanda stream tcp nowait amandabackup /usr/libexec/amanda/amandad amandad -auth=bsdtcp amdump amindexd amidxtaped </pre> Kick inetd to reload its config <pre> root@amdbkp-01:~# ps -ef | grep inetd root@amdbkp-01:~# kill -HUP 28822 </pre> Check that inetd picked up the config <pre> root@amdbkp-01:~# netstat -anp | grep 1008 tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 28822/inetd </pre> =====Xinetd way===== The packages built by Amanda team are dependant on xinet and unless you like some pain, you better stick to the defaults <pre> root@amdbkp-01:~# apt-get install gettext xinetd root@amdbkp-01:~# dpkg -i ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb </pre> Fix permissions on the directories that were owned by Ubuntu's own Amanda package backup:backup <pre> root@amdbkp-01:~# chown -R amandabackup:disk /var/lib/amanda root@amdbkp-01:~# chown -R amandabackup:disk /etc/amanda root@amdbkp-01:~# chown -R amandabackup:disk /tmp/amanda root@amdbkp-01:~# chown -R amandabackup:disk /amanda </pre> Amanda 3.3.5 .amandahosts is not /etc ---> /var/lib/amanda/.amandahosts <pre> amandabackup@amdbkp-01:~$ ls -l /var/lib/amanda/.amandahosts -rw------- 1 amandabackup disk 250 Jul 1 01:14 /var/lib/amanda/.amandahosts amandabackup@amdbkp-01:~$ cat /var/lib/amanda/.amandahosts localhost root amindexd amidxtaped localhost.localdomain root amindexd amidxtaped localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump amdbkp-01 amandabackup amdump </pre> ====Time==== Fix /etc/ntp.conf to point to internal NTP servers 192.168.0.10 and 192.168.1.10 <pre> driftfile /var/lib/ntp/ntp.drift statistics loopstats peerstats clockstats filegen loopstats file loopstats type day enable filegen peerstats file peerstats type day enable filegen clockstats file clockstats type day enable server 192.168.1.10 server 192.168.0.10 server ntp.ubuntu.com restrict -4 default kod notrap nomodify nopeer noquery restrict -6 default kod notrap nomodify nopeer noquery restrict 127.0.0.1 restrict ::1 </pre> <pre> ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *192.168.1.10 193.1.219.116 2 u 13 64 37 0.169 10.502 1.580 192.168.0.10 91.189.94.4 3 u 11 64 37 0.119 5.908 1.268 91.189.94.4 193.79.237.14 2 u 13 64 37 13.113 5.955 1.247 </pre> ====Holding space==== Create logical volume and filesystem: <pre> root@amdbkp-01:~# lvcreate -n amandalv -L100G amdbkp-01-vg root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/amandalv root@amdbkp-01:~# mount /dev/amdbkp-01-vg/amandalv /amanda root@amdbkp-01:~# chown amandabackup:disk /amanda root@amdbkp-01:~# chmod g+w /amanda </pre> Add the new filesystem to /etc/fstab <pre> root@amdbkp-01:~# grep amanda /etc/fstab /dev/mapper/amdbkp--01--vg-amandalv /amanda ext4 defaults 0 2 </pre> ====Postfix configuration==== We only need to send emails out, not to receive any, so the configuration is as slim as it can be: <pre> root@amdbkp-01:~# cat /etc/postfix/main.cf relayhost = smtp.production transport_maps = hash:/etc/postfix/transport root@amdbkp-01:~# cat /etc/postfix/transport production smtp:smtp.production corporate smtp:smtp.production root@amdbkp-01:~# cat /etc/aliases # See man 5 aliases for format postmaster: root root: alex.markelov @corporate </pre> Start Postfix <pre> root@amdbkp-01:~# service postfix start </pre> ====Configuring encryption==== Create encryption key <pre> root@amdbkp-01:~# pwgen -s 45 -N1 </pre> Save the key in <tt>/var/lib/amanda/.am_passphrase</tt> <pre> root@amdbkp-01:~# vi /var/lib/amanda/.am_passphrase root@amdbkp-01:~# chown amandabackup:disk /var/lib/amanda/.am_passphrase root@amdbkp-01:~# chmod 0600 /var/lib/amanda/.am_passphrase root@amdbkp-01:~# ls -l ~amandabackup/.am_passphrase -rw------- 1 amandabackup disk 77 Jul 1 00:46 /var/lib/amanda/.am_passphrase </pre> Dumptype tar-client-encrypt is responsible for encrypted backups. An example below makes full backups every time. <pre> define dumptype tar-client-encrypt { always-full-tar comment "partitions dumped with tar + encrypted" encrypt client client_encrypt "/usr/sbin/amcrypt-ossl" client_decrypt_option "-d" } </pre> ====Wrapper script to report backup status to Logstash/Kibana==== Below is the script that engages <tt>amdump</tt> for given configuration and then sends report in JSON format to Redis. <pre> # # Wrapper script to run dump and pump report out to Logstash/Kibana # # Pre-requisites: mlogger, RELP configuration in rsyslog # PATH=/usr/sbin:/usr/bin:/bin LOG_PRG_NAME=amanda_report if [ $# -lt 1 ] then echo "Usage: `basename $0` <backup_configuration_name>" exit -1 fi # Perform the backup amdump $1 # send out JSON formatted report to Logstash amreport --format=json, $1 | sed -e 's/^$/ /g' | tr -d '\n' | mlogger -t ${LOG_PRG_NAME} </pre> When you create cron job for backing up, use the script (example below) <pre> # Data Integration backups 0 20 * * * /etc/amanda/amdump_and_report.sh prod-1week-vtape-RAIT </pre> ===Tape library=== Changer is available as <tt>/dev/sg9</tt> device: <pre> root@amdbkp-01:~# mtx -f /dev/sg9 status Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export ) Data Transfer Element 0:Empty Storage Element 1:Full Storage Element 2:Full Storage Element 3:Full Storage Element 4:Full Storage Element 5:Empty Storage Element 6:Empty Storage Element 7:Empty Storage Element 8:Empty Storage Element 9:Empty Storage Element 10:Empty Storage Element 11:Empty Storage Element 12:Empty Storage Element 13:Empty Storage Element 14:Empty Storage Element 15:Empty Storage Element 16:Empty </pre> You can load particular tape using <tt>mtx</tt> command: <pre> root@amdbkp-01:~# mtx -f /dev/sg9 load 1 Loading media from Storage Element 1 into drive 0...done root@amdbkp-01:~# mtx -f /dev/sg9 status Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 1 Loaded) Storage Element 1:Empty Storage Element 2:Full Storage Element 3:Full Storage Element 4:Full Storage Element 5:Empty Storage Element 6:Empty Storage Element 7:Empty Storage Element 8:Empty Storage Element 9:Empty Storage Element 10:Empty Storage Element 11:Empty Storage Element 12:Empty Storage Element 13:Empty Storage Element 14:Empty Storage Element 15:Empty Storage Element 16:Empty </pre> Checking tape status using <tt>mt</tt> command: <pre> root@amdbkp-01:~# mt -f /dev/nst0 status drive type = 114 drive status = 1140850688 sense key error = 0 residue count = 0 file number = 0 block number = 0 </pre> ====Preparing tapes==== ====Test backup==== ====Test restore==== ====Test encryption==== ===Amazon S3=== Reference http://wiki.zmanda.com/index.php/How_To:Backup_to_Amazon_S3 Levente created Access KeyID and Secret Access Key for me Logged in to S3 console and created bucket called infra there. [[File:infra bucket.png]] In the bucket created folder called Amanda where the vTapes will reside. [[File:Amanda folder.png]] Add the server itself to allowed hosts <pre> root@amdbkp-01:~# cat /etc/amandahosts localhost amandabackup localhost root amindexd amidxtaped amdbkp-01 amandabackup amdbkp-01.production amandabackup </pre> Create production-S3 config <pre> root@amdbkp-01:~# su - amandabackup amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3 amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/log amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/index </pre> Below is content of configuration file /etc/amanda/production-S3/amanda.conf *WARNING* S3 secret key is not the real one in the config. <pre> org " production" # your organization name for reports mailto "root alex.markelov @corporate" # space separated list of operators at your site dumpuser "amandabackup" # the user to run dumps under inparallel 4 # maximum dumpers that will run in parallel (max 63) # this maximum can be increased at compile-time, # modifying MAX_DUMPERS in server-src/driverio.h dumporder "sssS" # specify the priority order of each dumper # s -> smallest size # S -> biggest size # t -> smallest time # T -> biggest time # b -> smallest bandwitdh # B -> biggest bandwitdh # try "BTBTBTBTBTBT" if you are not holding # disk constrained taperalgo first # The algorithm used to choose which dump image to send # to the taper. # Possible values: # [first|firstfit|largest|largestfit|smallest|last] # Default: first. # first First in - first out. # firstfit The first dump image that will fit on # the current tape. # largest The largest dump image. # largestfit The largest dump image that will fit on # the current tape. # smallest The smallest dump image. # last Last in - first out. displayunit "k" # Possible values: "k|m|g|t" # Default: k. # The unit used to print many numbers. # k=kilo, m=mega, g=giga, t=tera netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per sec dumpcycle 1 weeks # the number of days in the normal dump cycle runspercycle 5 # the number of amdump runs in dumpcycle days # (4 weeks * 5 amdump runs per week -- just weekdays) tapecycle 15 tapes # the number of tapes in rotation # 4 weeks (dumpcycle) times 5 tapes per week (just # the weekdays) plus a few to handle errors that # need amflush and so we do not overwrite the full # backups performed at the beginning of the previous # cycle usetimestamps yes # Default: Yes. This option allows Amanda to track # multiple runs per calendar day. The only reason one # might disable it is that Amanda versions before 2.5.1 # can't read logfiles written when this option was # enabled. bumpsize 20 Mb # minimum savings (threshold) to bump level 1 -> 2 bumppercent 20 # minimum savings (threshold) to bump level 1 -> 2 bumpdays 1 # minimum days at each level bumpmult 4 # threshold = bumpsize * bumpmult^(level-1) runtapes 1 # number of tapes to be used in a single run of amdump # amazonaws S3 device_property "S3_BUCKET_LOCATION" "eu-west-1" # Your S3 bucket location device_property "S3_ACCESS_KEY" "XXX" # Your S3 Access Key device_property "S3_SECRET_KEY" "XXX" # Your S3 Secret Key device_property "S3_SSL" "YES" # Curl needs to have S3 Certification Authority (Verisign today) # in its CA list. If connection fails, try setting this no NO tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09,10}" # Number of tapes in your "tapecycle" changerfile "s3-statefile" # Amanda will create this file tapetype S3 define tapetype S3 { comment "S3 Bucket" length 30 gigabytes # Bucket size 30GB } maxdumpsize -1 # Maximum number of bytes the planner will schedule # for a run (default: runtapes * tape_length). labelstr "^PRODUCTION-[0-9][0-9]*$" # label constraint regex: all tapes must match holdingdisk hd1 { comment "main holding disk" directory "/amanda" # where the holding disk is use -100 Mb # how much space can we use on it # a non-positive value means: # use all space but that value chunksize 1Gb # size of chunk if you want big dump to be # dumped on multiple files on holding disks # N Kb/Mb/Gb split images in chunks of size N # The maximum value should be # (MAX_FILE_SIZE - 1Mb) # 0 same as INT_MAX bytes } autoflush yes # # if autoflush is set to yes, then amdump will schedule all dump on # holding disks to be flush to tape during the run. infofile "/etc/amanda/production-S3/curinfo" # database DIRECTORY logdir "/etc/amanda/production-S3/log" # log directory indexdir "/etc/amanda/production-S3/index" # index directory define dumptype global { comment "Global definitions" # This is quite useful for setting global parameters, so you don't have # to type them everywhere. All dumptype definitions in this sample file # do include these definitions, either directly or indirectly. # There's nothing special about the name `global'; if you create any # dumptype that does not contain the word `global' or the name of any # other dumptype that contains it, these definitions won't apply. # Note that these definitions may be overridden in other # dumptypes, if the redefinitions appear *after* the `global' # dumptype name. # You may want to use this for globally enabling or disabling # indexing, recording, etc. Some examples: # index yes # record no # split_diskbuffer "/raid/amanda" # fallback_splitsize 64m } define dumptype always-full { global comment "Full dump of this filesystem always" compress none priority high dumpcycle 0 exclude list ".exclude.lst" } define dumptype always-full-tar { global program "GNUTAR" comment "Full backup with tar of this filesystem always" compress none priority high index dumpcycle 0 holdingdisk required exclude list ".exclude.lst" } define dumptype tar-client-encrypt { always-full-tar comment "partitions dumped with tar + encrypted" encrypt client client_encrypt "/usr/sbin/amcrypt-ossl" client_decrypt_option "-d" } define interface local { comment "a local disk" use 1000 kbps } define interface le0 { comment "10 Mbps ethernet" use 400 kbps } </pre> ====Creating vTapes==== Create first 9 vTapes in S3 and label it <pre> amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done; </pre> Check slot-09 vTape <pre> amandabackup@amdbkp-01:~$ amdevcheck production-S3 s3:infra/Amanda/production-S3/slot-09 </pre> The command should return SUCCESS if everything is OK ====Test backup==== Add /etc to disklist for test purpose: <pre> amandabackup@amdbkp-01:~$ cat /etc/amanda/production-S3/disklist amdbkp-01 /etc tar-client-encrypt </pre> Back it up: <pre> amandabackup@amdbkp-01:~$ amdump production-S3 & [1] 28522 backup@amdbkp-01:~$ amstatus production-S3 Using /etc/amanda/production-S3/log/amdump.1 From Mon Jun 30 23:58:09 IST 2014 amdbkp-01:/etc 0 3251k finished (23:58:11) SUMMARY part real estimated size size partition : 1 estimated : 1 3240k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 3251k 3240k (100.34%) (100.34%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 1 3251k 3240k (100.34%) (100.34%) tape 1 : 1 3251k 3240k ( 0.01%) PRODUCTION-001 (2 chunks) 4 dumpers idle : no-dumpers taper status: Idle taper qlen: 0 network free kps: 600 holding space : 97673216k (100.00%) chunker0 busy : 0:00:00 ( 14.23%) dumper0 busy : 0:00:00 ( 13.83%) taper busy : 0:00:01 ( 42.72%) 0 dumpers busy : 0:00:02 ( 86.13%) not-idle: 0:00:01 ( 50.26%) no-dumpers: 0:00:01 ( 49.74%) 1 dumper busy : 0:00:00 ( 13.87%) [1]+ Done amdump production-S3 </pre> Check S3 web UI to see that the tape PRODUCTION-001 was used [[File:production-S3 vtapes.png]] ====Test restore==== You need to make sure your /etc/amanda/amanda-client.conf doesn't have tapedev defined. It led to error for S3 when I was trying to set it to what I believed was correct (tapedev "s3:infra/Amanda" or tapedev "s3:infra/Amanda/production-S3"). Error I was getting "Amanda header not found -- unlabeled volume?" <pre> root@amdbkp-01:~# amrecover production-S3 AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-01) 200 Working date set to 2014-07-01. 200 Config set to production-S3. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> setdisk /etc 200 Disk set to /etc. amrecover> lcd /tmp amrecover> add inetd.conf Added file /inetd.conf amrecover> extract Extracting files using tape drive s3:infra/Amanda on host localhost. The following tapes are needed: PRODUCTION-002 Extracting files using tape drive s3:infra/Amanda on host localhost. Load tape PRODUCTION-002 now Continue [?/Y/n/s/d]? y Amanda header not found -- unlabeled volume? Load tape PRODUCTION-002 now Continue [?/Y/n/d]? n Got no header and data from server, check in amidxtaped.*.debug and amandad.*.debug files on server amrecover> quit 200 Good bye. </pre> Commenting out tapedev in amanda-client.conf solved the issue. Ref: https://forums.zmanda.com/showthread.php?2811-Problem-amrecover-from-S3 <pre> root@amdbkp-01:/tmp/1# cat /etc/amanda/amanda-client.conf # # amanda-client.conf - sample Amanda client configuration file. # # This file normally goes in /etc/amanda/amanda-client.conf. # conf "DailySet1" # your config name index_server "localhost" # your amindexd server tape_server "localhost" # your amidxtaped server #tapedev "s3:infra/Amanda" #tapedev "tape:/dev/YOUR-TAPE-DEVICE-HERE" # your tape device # if not set, Use configure or ask server. # if set to empty string "", ask server # amrecover will use the changer if set to the value # of 'amrecover_changer' in the server amanda.conf. # auth - authentication scheme to use between server and client. # Valid values are "bsd", "bsdudp", "bsdtcp", "krb5", "local", # "rsh" and "ssh". # Default: [auth "bsdtcp"] auth "bsdtcp" ssh_keys "" # your ssh keys file if you use ssh auth </pre> Successful restore after fixing /etc/amanda/amanda-client.conf. We will create /tmp/1 to restore file into. <pre> root@amdbkp-01:/tmp# mkdir /tmp/1 root@amdbkp-01:/tmp# amrecover production-S3 AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-01) 200 Working date set to 2014-07-01. 200 Config set to production-S3. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> lcd /tmp/1 amrecover> setdisk /etc 200 Disk set to /etc. amrecover> add inetd.conf Added file /inetd.conf amrecover> extract Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost. The following tapes are needed: PRODUCTION-002 Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost. Load tape PRODUCTION-002 now Continue [?/Y/n/s/d]? y Restoring files into directory /tmp/1 All existing files in /tmp/1 can be deleted Continue [?/Y/n]? y ./inetd.conf amrecover> quit 200 Good bye. </pre> Now checking the file we just restored: <pre> root@amdbkp-01:/tmp# cd 1 root@amdbkp-01:/tmp/1# ls -al total 12 drwxr-xr-x 2 root root 4096 Jul 1 01:45 . drwxrwxrwt 6 root root 4096 Jul 1 01:45 .. -rw-r--r-- 1 root root 1170 Jul 1 01:12 inetd.conf </pre> ====Test encryption==== *NOTE* You can use the same method to download vTape from S3 and restore locally without using Amanda To prove that we have the vTapes encrypted at S3, first, download vTape from S3 using web UI [[File:sample tape download.png]] <pre> Alexs-MacBook:Downloads alex$ ls -l slot-02f00000001-b0000000000000000.data -rw-r--r--@ 1 alex staff 3512368 1 Jul 01:16 slot-02f00000001-b0000000000000000.data </pre> Copy the file over to Amanda server <pre> Alexs-MacBook:Downloads alex$ scp slot-02f00000001-b0000000000000000.data amdbkp-01:/tmp/ slot-02f00000001-b0000000000000000.data 100% 3430KB 3.4MB/s 00:01 </pre> Check the vTape with and without decryption <pre> root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |/usr/sbin/amcrypt-ossl -d|tar tvf - drwxr-xr-x root/root 1865 2014-07-01 01:13 ./ drwxr-xr-x root/root 6 2014-04-24 15:25 ./X11/ drwxr-xr-x root/root 1 2012-07-03 07:21 ./X11/xkb/ drwxr-xr-x root/root 22 2014-04-24 15:32 ./acpi/ drwxr-xr-x root/root 11 2014-04-24 15:32 ./acpi/events/ drwxr-xr-x root/root 952 2014-06-11 06:36 ./alternatives/ drwxrwx--- amandabackup/disk 53 2014-07-01 00:46 ./amanda/ <more output skipped> </pre> Now, try without decryption part of the command (/usr/sbin/amcrypt-ossl -d) and see how tar can't do anything with the tape: <pre> root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |tar tvf - tar: This does not look like a tar archive tar: Skipping to next header 107+1 records in 107+1 records out tar: 3512368 bytes (3.5 MB) copiedExiting with failure status due to previous errors, 0.00662878 s, 530 MB/s </pre> ===NAS based vTapes=== The below based on [http://wiki.zmanda.com/index.php/How_To:Set_Up_Virtual_Tapes vTape setup in Amanda wiki]. We need NFS client <pre> root@amdbkp-01:~# apt-get install nfs-common </pre> Mount the test NFS share and prep directory for vTapes <pre> root@amdbkp-01:~# mount.nfs dc-nas-01:/test /vtapes root@amdbkp-01:~# mkdir /vtapes/amanda.vtapes root@amdbkp-01:~# chown amandabackup:disk /vtapes/amanda.vtapes root@amdbkp-01:~# ls -dl /vtapes/amanda.vtapes drwxrwx---+ 2 amandabackup disk 4096 Jul 7 10:50 /vtapes/amanda.vtapes </pre> ====Creating vTapes==== Switch to amanda user <pre> root@amdbkp-01:~# su - amandabackup </pre> Prepare configuration directory. We will take S3 directory and files as the starting point: <pre> amandabackup@amdbkp-01:~$ cd /etc/amanda/ amandabackup@amdbkp-01:~$ cp -rp production-S3 production-vtape amandabackup@amdbkp-01:~$ cp -rp production-S3 production-vtape amandabackup@amdbkp-01:~$ cd production-vtape/ amandabackup@amdbkp-01:~$ rm -rf tapelist* disklist s3-statefile curinfo index/* log/* </pre> Define tape type VTAPE and make the size the same as S3 bucket, because we are going to do RAIT with S3 later. <pre> tapetype VTAPE define tapetype VTAPE { comment "NFS vTape share" length 30 gigabytes # Bucket size 30GB } </pre> Define tpchanger <pre> tpchanger "chg-disk:/vtapes/amanda.vtapes" changerfile "vtapes-statefile" # Amanda will create this file </pre> Create the directory structure and label the first 9 tapes <pre> amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done; </pre> amdevcheck failed until the first backup run. Must be a bug. <pre> amandabackup@amdbkp-01:~$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1 MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory DEVICE_ERROR </pre> ====Test backup==== Check which tape is next in the backup cycle <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amadmin production-vtape tape next </pre> Start the backup (note: we had copied the configuration from S3 configuration directory, so it's already has disklist file with /etc in there) and check its status: <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdump production-vtape & [1] 10546 amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amstatus production-vtape Using /etc/amanda/production-vtape/log/amdump.1 From Mon Jul 7 12:26:13 IST 2014 amdbkp-01:/etc 0 3361k finished (12:26:14) SUMMARY part real estimated size size partition : 1 estimated : 1 3350k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 3361k 3350k (100.33%) (100.33%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 1 3361k 3350k (100.33%) (100.33%) tape 1 : 1 3360k 3360k ( 0.01%) PRODUCTION-001 (1 chunks) 4 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 600 holding space : 97673216k (100.00%) chunker0 busy : 0:00:00 ( 10.25%) dumper0 busy : 0:00:00 ( 9.59%) taper busy : 0:00:00 ( 8.04%) 0 dumpers busy : 0:00:01 ( 90.41%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:00 ( 9.59%) [1]+ Done amdump production-vtape </pre> Now when you do amdevcheck on the slot1 it fails, but succeeds on vtape directory above, because that's where the data file is. Strange! <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1 MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory DEVICE_ERROR amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes SUCCESS </pre> ====Test restore==== Prepare destination directory where we restore into for the test: <pre> root@amdbkp-01:~# mkdir /tmp/1 && cd /tmp </pre> Run the test restore <pre> root@amdbkp-01:/tmp# amrecover production-vtape AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-07) 200 Working date set to 2014-07-07. 200 Config set to production-vtape. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> lcd /tmp/1 amrecover> setdisk /etc 200 Disk set to /etc. amrecover> add inetd.conf Added file /inetd.conf amrecover> extract Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost. The following tapes are needed: PRODUCTION-001 Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost. Load tape PRODUCTION-001 now Continue [?/Y/n/s/d]? Restoring files into directory /tmp/1 All existing files in /tmp/1 can be deleted Continue [?/Y/n]? ./inetd.conf amrecover> quit 200 Good bye. </pre> And now check the file we just restored: <pre> root@amdbkp-01:/tmp# vi 1/inetd.conf </pre> ====Test encryption==== Locate the vTape file. This is 00001.amdbkp-01._etc.0 in our case: <pre> root@amdbkp-01:/tmp# ls /vtapes/amanda.vtapes/slot1/ 00000.PRODUCTION-001 00001.amdbkp-01._etc.0 root@amdbkp-01:/tmp# ls -l /vtapes/amanda.vtapes/slot1/ total 3432 -rw-rw----+ 1 amandabackup disk 32768 Jul 7 12:26 00000.PRODUCTION-001 -rw-rw----+ 1 amandabackup disk 3473520 Jul 7 12:26 00001.amdbkp-01._etc.0 </pre> Use dd to inspect the header of the vTape (note: we need only the header hence bs=720) <pre> root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=720 count=1 AMANDA: SPLIT_FILE 20140707122613 amdbkp-01 /etc part 1/-1 lev 0 comp N program /bin/tar crypt enc client_encrypt /usr/sbin/amcrypt-ossl client_decrypt_option -d ORIGSIZE=3360 DLE=<<ENDDLE <dle> <program>GNUTAR</program> <disk>/etc</disk> <level>0</level> <auth>BSDTCP</auth> <encrypt>CUSTOM<custom-encrypt-program>/usr/sbin/amcrypt-ossl</custom-encrypt-program> <decrypt-option>-d</decrypt-option> </encrypt> <record>YES</record> <index>YES</index> <datapath>AMANDA</datapath> <exclude> <list>.exclude.lst</list> </exclude> </dle> ENDDLE To restore, position tape at start of file and run: dd if=<tape> bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -xpGf - ... 1+0 records in 1+0 records out 720 bytes (720 B) copied, 0.000256799 s, 2.8 MB/s </pre> Test the vTape by extracting the list of files inside: <pre> root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -tvf - drwxr-xr-x root/root 1920 2014-07-07 10:50 ./ drwxr-xr-x root/root 6 2014-04-24 15:25 ./X11/ drwxr-xr-x root/root 1 2012-07-03 07:21 ./X11/xkb/ drwxr-xr-x root/root 22 2014-04-24 15:32 ./acpi/ drwxr-xr-x root/root 11 2014-04-24 15:32 ./acpi/events/ drwxr-xr-x root/root 952 2014-06-11 06:36 ./alternatives/ drwxrwx--- amandabackup/disk 71 2014-07-07 11:04 ./amanda/ <… more output skipped…> </pre> Doing the same without decryption (/usr/sbin/amcrypt-ossl -d) will result in failure: <pre> root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /bin/tar -tvf - /bin/tar: This does not look like a tar archive /bin/tar: Skipping to next header 105+1 records in 105+1 records out /bin/tar: 3440752 bytes (3.4 MB) copiedExiting with failure status due to previous errors , 0.0067797 s, 508 MB/s </pre> ===RAIT (NAS + S3)=== Here is the part of Amnda configuration file amanda.conf, which defines the RAIT. *WARNING* tapetype should not be inside each changer definition! I took S3 tapetype as the one here, but it can be called whatever. <pre> tpchanger "chg-rait:{nas,s3}" tapetype S3 define changer nas { # vTapes tpchanger "chg-disk:/vtapes/amanda.vtapes" changerfile "vtapes-statefile" # Amanda will create this file } define changer s3 { # amazonaws S3 device_property "S3_BUCKET_LOCATION" "eu-west-1" # Your S3 bucket location device_property "S3_ACCESS_KEY" "XXX" # Your S3 Access Key device_property "S3_SECRET_KEY" "XXX" # Your S3 Secret Key device_property "S3_SSL" "YES" # Curl needs to have S3 Certification Authority (Verisign today) # in its CA list. If connection fails, try setting this no NO tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09}" # Number of tapes in your "tapecycle" changerfile "s3-statefile" # Amanda will create this file } define tapetype S3 { comment "S3 Bucket" length 30 gigabytes # Bucket size 30GB blocksize 1 megabytes } </pre> tapelist we take from S3 configuration, because RAIT driver won't create tapes for us in S3. Took the list from S3 configuration directory. Mind the blocksize parameter, it's set in amanda.conf file. <pre> 0 PRODUCTION-009 reuse BLOCKSIZE:10240 0 PRODUCTION-008 reuse BLOCKSIZE:10240 0 PRODUCTION-007 reuse BLOCKSIZE:10240 0 PRODUCTION-006 reuse BLOCKSIZE:10240 0 PRODUCTION-005 reuse BLOCKSIZE:10240 0 PRODUCTION-004 reuse BLOCKSIZE:10240 0 PRODUCTION-003 reuse BLOCKSIZE:10240 0 PRODUCTION-002 reuse BLOCKSIZE:10240 0 PRODUCTION-001 reuse BLOCKSIZE:10240 </pre> ====Creating vTapes==== Because we had tested vTape and S3 configurations by now, we have tapes in the tow locations having different timestamps and data in it, so RAIT check will be failing. To fix this we need to clean up the both tape locations and re-create tapes. Removing and re-creating vTapes <pre> amandabackup@amdbkp-01:~$ rm -rf /vtapes/amanda.vtapes/* amandabackup@amdbkp-01:~$ cd /etc/amanda/production-vtape amandabackup@amdbkp-01:~$ rm -rf tapelist* vtapes-statefile curinfo index/* log/* amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done; </pre> Removing and re-creating vTapes in S3 <pre> amandabackup@amdbkp-01:~$ cd /etc/amanda/production-S3 amandabackup@amdbkp-01:~$ rm -rf tapelist* s3-statefile curinfo index/* log/* </pre> Login to Amazon S3 management console and remove all tape files from infra/Amanda/production-S3 bucket, then re-create the tapes from the backup server: <pre> amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done; </pre> Copy production-vtape/vtapes-statefile and production-S3/s3-statefile to /etc/amanda/production-S3-vtape-RAIT <pre> amandabackup@amdbkp-01:~$ cat s3-statefile $STATE = { 'current_slot_by_config' => { 'production-S3-vtape-RAIT' => 's3:infra/Amanda/production-S3/slot-04' }, 'slots' => { 's3:infra/Amanda/production-S3/slot-09' => { 'label' => 'PRODUCTION-009', 'device_status' => '0', 'f_type' => '1', 'state' => 1, 'device_error' => undef }, 's3:infra/Amanda/production-S3/slot-03' => { 'label' => 'PRODUCTION-003', 'device_status' => '0', 'f_type' => '1', 'state' => 1, 'device_error' => undef }, 's3:infra/Amanda/production-S3/slot-04' => { 'device_status' => '0', 'label' => 'PRODUCTION-004', 'f_type' => '1', 'device_error' => undef, 'state' => 1 }, 's3:infra/Amanda/production-S3/slot-07' => { 'label' => 'PRODUCTION-007', 'device_status' => '0', 'f_type' => '1', 'state' => 1, 'device_error' => undef }, 's3:infra/Amanda/production-S3/slot-01' => { 'label' => 'PRODUCTION-001', 'device_status' => '0', 'f_type' => '1', 'state' => 1, 'device_error' => undef }, 's3:infra/Amanda/production-S3/slot-06' => { 'device_status' => '0', 'label' => 'PRODUCTION-006', 'f_type' => '1', 'device_error' => undef, 'state' => 1 }, 's3:infra/Amanda/production-S3/slot-08' => { 'device_status' => '0', 'label' => 'PRODUCTION-008', 'f_type' => '1', 'device_error' => undef, 'state' => 1 }, 's3:infra/Amanda/production-S3/slot-05' => { 'label' => 'PRODUCTION-005', 'device_status' => '0', 'f_type' => '1', 'state' => 1, 'device_error' => undef }, 's3:infra/Amanda/production-S3/slot-02' => { 'device_status' => '0', 'label' => 'PRODUCTION-002', 'f_type' => '1', 'device_error' => undef, 'state' => 1 } } }; </pre> <pre> amandabackup@amdbkp-01:~$ cat vtapes-statefile $STATE = { 'drives' => { '/vtapes/amanda.vtapes/drive0' => {} } }; </pre> If we need more tapes in the RAIT, [http://www.bic.mni.mcgill.ca/PersonalMalouinjeanfrancois/AmandaHowtoAddVirtualTapesRait this article] describes how to add more tapes. This requires fiddling with configuration file. ====Test backup==== Run test backup and check its status: <pre> amandabackup@amdbkp-01:~$ amdump production-S3-vtape-RAIT & [1] 11922 amandabackup@amdbkp-01:~$ amstatus production-S3-vtape-RAIT Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1 From Mon Jul 7 19:55:30 IST 2014 amdbkp-01:/etc 0 3421k finished (19:55:31) SUMMARY part real estimated size size partition : 1 estimated : 1 3420k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 3421k 3420k (100.03%) (100.03%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 1 3421k 3420k (100.03%) (100.03%) tape 1 : 1 3420k 3420k ( 0.01%) PRODUCTION-002 (1 chunks) 4 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 600 holding space : 97673216k (100.00%) chunker0 busy : 0:00:00 ( 11.74%) dumper0 busy : 0:00:00 ( 10.84%) taper busy : 0:00:00 ( 54.68%) 0 dumpers busy : 0:00:01 ( 89.16%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:00 ( 10.84%) </pre> Now check that both locations were used. Check S3 and NAS and see that tape 02 was used. S3: [[File:RAIT tape 02 on S3 used.png]] NAS: [[File:RAIT tape 02 on NAS used.png]] ====Test restore==== Prepare test restore destination directory and restore a file into it: <pre> root@amdbkp-01:~# mkdir /tmp/1 root@amdbkp-01:~# cd /tmp/1 root@amdbkp-01:/tmp/1# amrecover production-S3-vtape-RAIT AMRECOVER Version 3.3.5. Contacting server on localhost ... 220 amdbkp-01 AMANDA index server (3.3.5) ready. Setting restore date to today (2014-07-07) 200 Working date set to 2014-07-07. 200 Config set to production-S3-vtape-RAIT. 200 Dump host set to amdbkp-01. Use the setdisk command to choose dump disk to recover amrecover> setdisk /etc 200 Disk set to /etc. amrecover> lcd /tmp/1 amrecover> add resolv.conf Added file /resolv.conf amrecover> extract Extracting files using tape drive chg-rait:{nas,s3} on host localhost. The following tapes are needed: PRODUCTION-002 Extracting files using tape drive chg-rait:{nas,s3} on host localhost. Load tape PRODUCTION-002 now Continue [?/Y/n/s/d]? Restoring files into directory /tmp/1 All existing files in /tmp/1 can be deleted Continue [?/Y/n]? ./resolv.conf amrecover> quit 200 Good bye. </pre> Check what we have restored: <pre> root@amdbkp-01:/tmp/1# ls -l total 0 lrwxrwxrwx 1 root root 29 Apr 24 15:25 resolv.conf -> ../run/resolvconf/resolv.conf </pre> ====Test encryption==== It's the same process as what we did for S3 and NAS respectively above. Fetch the vTape file from S3 and use dd to inspect it and you can do the same with NAS based vTape file. ===RAIT (NAS Datacenter + NAS off-site)=== We will create a RAIT over two NAS appliances: * Office comms room office-nas-01 * Datacenter dc-nas-01 From the previous configuration (NAS+S3) we already have vTapes share mounted off the NAS in Datacenter. <pre> root@amdbkp-01:~# df -k /vtapes Filesystem 1K-blocks Used Available Use% Mounted on dc-nas-01:/test 29056019968 9221493632 19834002048 32% /vtapes </pre> Now we mount NFS share off office based (off-site) NAS: <pre> root@amdbkp-01:~# mkdir /vtapes-offsite root@amdbkp-01:~# mount office-nas-01:/AMANDA /vtapes-offsite root@amdbkp-01:~# df -k | grep vtapes Filesystem 1K-blocks Used Available Use% Mounted on dc-nas-01:/test 29056019968 9222531968 19832963712 32% /vtapes office-nas-01:/AMANDA 5763950016 4348221728 1415204000 76% /vtapes-offsite </pre> Added entries to stab <pre> root@amdbkp-01:~# tail -2 /etc/fstab dc-nas-01:/test /vtapes nfs defaults 2 2 office-nas-01:/AMANDA /vtapes-offsite nfs defaults 2 2 </pre> Fixing ownership and permissions on the mounted share: <pre> root@amdbkp-01:~# chown -R amandabackup:disk /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxrwx 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxrwx 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite root@amdbkp-01:~# chmod o-w /vtapes-offsite root@amdbkp-01:~# ls -ld /vtapes* drwxrwxrwx+ 6 root root 4096 Jul 14 20:28 /vtapes drwxrwxr-x 2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite </pre> Create new Amanda configuration. S3+NAS RAIT is the basis for this one. <pre> root@amdbkp-01:~# su - amandabackup amandabackup@amdbkp-01:~$ cd /etc/amanda amandabackup@amdbkp-01:/etc/amanda$ ls amanda-client.conf production-S3 production-S3-vtape-RAIT production-tape production-vtape amandabackup@amdbkp-01:/etc/amanda$ cp -rp production-S3-vtape-RAIT production-vtape-RAIT amandabackup@amdbkp-01:/etc/amanda$ cd production-vtape-RAIT/ amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l total 21 -rw-rw-r-- 1 amandabackup disk 7732 Jul 7 20:17 amanda.conf -rw------- 1 amandabackup disk 135 Jul 7 17:01 changer drwxr-xr-x 3 amandabackup disk 1024 Jul 7 19:48 curinfo -rw-r--r-- 1 amandabackup disk 45 Jul 7 12:11 disklist drwxrwxr-x 3 amandabackup disk 1024 Jul 7 19:48 index drwxrwxr-x 3 amandabackup disk 1024 Jul 7 20:57 log -rw------- 1 amandabackup disk 5784 Jul 7 22:44 s3-statefile -rw------- 1 amandabackup disk 472 Jul 7 20:57 tapelist -rw------- 1 amandabackup disk 0 Jul 7 19:48 tapelist.lock -rw------- 1 amandabackup disk 135 Jul 7 22:44 vtapes-statefile amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ mkdir /vtapes/amanda.vtapes-RAIT amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -ld /vtapes/amanda.vtapes-RAIT drwxrwx---+ 2 amandabackup disk 4096 Jul 16 16:42 /vtapes/amanda.vtapes-RAIT </pre> Clean up configuration directory off S3+NAS files <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ rm -rf tapelist* vtapes-statefile curinfo index/* log/* s3-statefile changer amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l total 11 -rw-rw-r-- 1 amandabackup disk 7732 Jul 7 20:17 amanda.conf -rw-r--r-- 1 amandabackup disk 45 Jul 7 12:11 disklist drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 index drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 log </pre> Edit Amanda file. We define the two changers and the RAIT changer based on the two: <pre> tpchanger "chg-rait:{nas-tc,nas-gqh}" tapetype VTAPE define changer nas-tc { tpchanger "chg-disk:/vtapes/amanda.vtapes-RAIT" changerfile "vtapes-statefile-tc" # Amanda will create this file } define changer nas-gqh { tpchanger "chg-disk:/vtapes-offsite" changerfile "vtapes-statefile-gqh" # Amanda will create this file } define tapetype VTAPE { comment "NFS vTape share" length 30 gigabytes # Bucket size 30GB } </pre> ====Creating vTapes==== Create vTapes and label it. The good thing about two NAS location is that you can do this in one go for both (not like with S3 and NAS, where you had to create vTapes for each configuration separately) <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes-RAIT/slot${i} /vtapes-offsite/slot${i} && amlabel production-vtape-RAIT PRODUCTION-00$i slot $i; done; Reading label... Found an empty tape. Writing label 'PRODUCTION-001'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-002'... Checking label... Success! <...skipped similar output for the rest of the vTapes...> </pre> Check the results in both NAS units. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/ /vtapes-offsite/ /vtapes-offsite/slot1 /vtapes-offsite/slot1/00000.PRODUCTION-001 /vtapes-offsite/slot8 /vtapes-offsite/slot8/00000.PRODUCTION-008 /vtapes-offsite/slot4 /vtapes-offsite/slot4/00000.PRODUCTION-004 /vtapes-offsite/slot3 /vtapes-offsite/slot3/00000.PRODUCTION-003 /vtapes-offsite/slot6 /vtapes-offsite/slot6/00000.PRODUCTION-006 /vtapes-offsite/slot2 /vtapes-offsite/slot2/00000.PRODUCTION-002 /vtapes-offsite/slot7 /vtapes-offsite/slot7/00000.PRODUCTION-007 /vtapes-offsite/slot5 /vtapes-offsite/slot5/00000.PRODUCTION-005 /vtapes-offsite/slot9 /vtapes-offsite/slot9/00000.PRODUCTION-009 amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/slot2 /vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002 /vtapes/amanda.vtapes-RAIT/slot5 /vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005 /vtapes/amanda.vtapes-RAIT/slot1 /vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001 /vtapes/amanda.vtapes-RAIT/slot4 /vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004 /vtapes/amanda.vtapes-RAIT/slot8 /vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008 /vtapes/amanda.vtapes-RAIT/slot7 /vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007 /vtapes/amanda.vtapes-RAIT/slot9 /vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009 /vtapes/amanda.vtapes-RAIT/slot6 /vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006 /vtapes/amanda.vtapes-RAIT/slot3 /vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003 </pre> Checking configuration. Ignore the error about /etc/.exclude.lst missing, we don't have it yet (and may not even need it) <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT Amanda Tape Server Host Check ----------------------------- Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB slot {1,1}: volume 'PRODUCTION-001' Will write to volume 'PRODUCTION-001' in slot {1,1}. NOTE: skipping tape-writable test Server check took 0.196 seconds Amanda Backup Client Hosts Check -------------------------------- ERROR: amdbkp-01: [Can't open exclude file /etc/.exclude.lst (No such file or directory)] Client check: 1 host checked in 2.066 seconds. 1 problem found. (brought to you by Amanda 3.3.5) </pre> ====Test backup==== <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amdump production-vtape-RAIT & [1] 11946 amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1 From Wed Jul 16 16:54:46 IST 2014 amdbkp-01:/etc 0 3561k finished (16:54:48) SUMMARY part real estimated size size partition : 1 estimated : 1 3560k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 3561k 3560k (100.03%) (100.03%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 1 3561k 3560k (100.03%) (100.03%) tape 1 : 1 3560k 3560k ( 0.01%) PRODUCTION-001 (1 chunks) 4 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 600 holding space : 97673216k (100.00%) chunker0 busy : 0:00:00 ( 23.85%) dumper0 busy : 0:00:00 ( 23.28%) taper busy : 0:00:00 ( 12.77%) 0 dumpers busy : 0:00:01 ( 76.63%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:00 ( 23.28%) [1]+ Done amdump production-vtape-RAIT </pre> Check the result (new tape file in slot1) in both NAS. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/ /vtapes-offsite/ /vtapes-offsite/slot1 /vtapes-offsite/slot1/00001.amdbkp-01._etc.0 /vtapes-offsite/slot1/00000.PRODUCTION-001 /vtapes-offsite/data /vtapes-offsite/slot8 /vtapes-offsite/slot8/00000.PRODUCTION-008 /vtapes-offsite/slot4 /vtapes-offsite/slot4/00000.PRODUCTION-004 /vtapes-offsite/slot3 /vtapes-offsite/slot3/00000.PRODUCTION-003 /vtapes-offsite/slot6 /vtapes-offsite/slot6/00000.PRODUCTION-006 /vtapes-offsite/slot2 /vtapes-offsite/slot2/00000.PRODUCTION-002 /vtapes-offsite/slot7 /vtapes-offsite/slot7/00000.PRODUCTION-007 /vtapes-offsite/slot5 /vtapes-offsite/slot5/00000.PRODUCTION-005 /vtapes-offsite/slot9 /vtapes-offsite/slot9/00000.PRODUCTION-009 amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/ /vtapes/amanda.vtapes-RAIT/slot2 /vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002 /vtapes/amanda.vtapes-RAIT/slot5 /vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005 /vtapes/amanda.vtapes-RAIT/slot1 /vtapes/amanda.vtapes-RAIT/slot1/00001.amdbkp-01._etc.0 /vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001 /vtapes/amanda.vtapes-RAIT/slot4 /vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004 /vtapes/amanda.vtapes-RAIT/slot8 /vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008 /vtapes/amanda.vtapes-RAIT/slot7 /vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007 /vtapes/amanda.vtapes-RAIT/slot9 /vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009 /vtapes/amanda.vtapes-RAIT/slot6 /vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006 /vtapes/amanda.vtapes-RAIT/data /vtapes/amanda.vtapes-RAIT/slot3 /vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003 </pre> Let's get some bigger files to test backup speed. There are some big TSM files in nas-02:/test, let's grub those. <pre> root@amdbkp-01:~# mount nas-02:/test /mnt </pre> We will need a separate filesystem for the test. Check free space left in the VG. <pre> root@amdbkp-01:~# vgdisplay --- Volume group --- VG Name amdbkp-01-vg System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 12 VG Access read/write VG Status resizable MAX LV 0 Cur LV 7 Open LV 7 Max PV 0 Cur PV 2 Act PV 2 VG Size 272.98 GiB PE Size 4.00 MiB Total PE 69883 Alloc PE / Size 36243 / 141.57 GiB Free PE / Size 33640 / 131.41 GiB VG UUID ENArtj-VebI-x5WG-Lpc0-zN5X-cLpL-FRsT9W </pre> Create new LV and filesystem. <pre> root@amdbkp-01:~# lvcreate -n testlv -L10G amdbkp-01-vg Logical volume "testlv" created root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/testlv root@amdbkp-01:~# mkdir /test root@amdbkp-01:~# mount /dev/amdbkp-01-vg/testlv /test root@amdbkp-01:~# chmod 0755 /test root@amdbkp-01:~# cp /mnt/TSM71/{TSM_7.1_WIN_*,TSM_7.1_LIN86_AGT_ML.bin} /test/ root@amdbkp-01:~# df -k /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/amdbkp--01--vg-testlv 10190136 4868312 4797536 51% /test </pre> Add /test to disklist. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ cat /etc/amanda/production-vtape-RAIT/disklist #amdbkp-01 /etc tar-client-encrypt amdbkp-01 /test tar-client-encrypt </pre> Check configuration for errors. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT Amanda Tape Server Host Check ----------------------------- Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB slot {1,1}: volume 'PRODUCTION-001' is still active and cannot be overwritten slot {2,2}: volume 'PRODUCTION-002' Will write to volume 'PRODUCTION-002' in slot {2,2}. NOTE: skipping tape-writable test NOTE: info dir /etc/amanda/production-S3-vtape-RAIT/curinfo/amdbkp-01/_test does not exist NOTE: it will be created on the next run. NOTE: index dir /etc/amanda/production-S3-vtape-RAIT/index/amdbkp-01/_test does not exist NOTE: it will be created on the next run. Server check took 0.220 seconds Amanda Backup Client Hosts Check -------------------------------- ERROR: amdbkp-01: [Can't open exclude file /test/.exclude.lst (No such file or directory)] Client check: 1 host checked in 1.095 seconds. 1 problem found. (brought to you by Amanda 3.3.5) </pre> Run dump of /test. I thought it was going to take some time and I'd have to walk away and check the result later, so I used 'nohup'. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ nohup amdump production-vtape-RAIT & </pre> Checking the status of the dump and it's impressively fast. Mind you, it's the percentage completed to holding disk (!) not the vTapes yet. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT Using /etc/amanda/production-S3-vtape-RAIT/log/amdump From Wed Jul 16 17:13:35 IST 2014 amdbkp-01:/test 0 4838040k dumping 1727328k ( 35.70%) (17:13:36) SUMMARY part real estimated size size partition : 1 estimated : 1 4838040k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 1 1727328k 4838040k ( 35.70%) ( 35.70%) dumped : 0 0k 0k ( 0.00%) ( 0.00%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 0 0k 0k ( 0.00%) ( 0.00%) 3 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 0 holding space : 92834976k ( 95.05%) dumper0 busy : 0:00:00 ( 0.00%) 0 dumpers busy : 0:00:01 (100.00%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:00 ( 0.00%) </pre> And now it's writing to vTapes. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT Using /etc/amanda/production-S3-vtape-RAIT/log/amdump From Wed Jul 16 17:13:35 IST 2014 amdbkp-01:/test 0 4838041k dump done, writing to tape (1241440k done (25.66%)) (17:14:17) SUMMARY part real estimated size size partition : 1 estimated : 1 4838040k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 4838041k 4838040k (100.00%) (100.00%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 1 4838041k 4838040k (100.00%) (100.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 0 0k 0k ( 0.00%) ( 0.00%) tape 1 : 0 0k 0k ( 0.00%) PRODUCTION-002 4 dumpers idle : 0 taper status: Writing amdbkp-01:/test taper qlen: 0 network free kps: 600 holding space : 92835015k ( 95.05%) chunker0 busy : 0:00:41 ( 97.37%) dumper0 busy : 0:00:41 ( 97.35%) 0 dumpers busy : 0:00:01 ( 2.65%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:41 ( 97.35%) 0: 0:00:41 (100.00%) </pre> Done in 41 seconds (!). What was strange though is that it seemed to have being writing to NAS still even after amstatus reported 100% completion. Need to test it again. When I run 'df -k' right after amstatus reported backup complete, it stayed there for about a minute before I saw df output. And at the same time observing inbound traffic to the office QNAP (via web UI System status -> Bandwidth usage), it was showing ~9MB/s inboud traffic even after amstatus reported 100% completion. <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus production-vtape-RAIT Using /etc/amanda/production-S3-vtape-RAIT/log/amdump From Wed Jul 16 17:13:35 IST 2014 amdbkp-01:/test 0 4838041k dump done, writing to tape (4838048k done (100.00%)) (17:14:17) SUMMARY part real estimated size size partition : 1 estimated : 1 4838040k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 0 0k ( 0.00%) dumping to tape : 0 0k ( 0.00%) dumping : 0 0k 0k ( 0.00%) ( 0.00%) dumped : 1 4838041k 4838040k (100.00%) (100.00%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 1 4838041k 4838040k (100.00%) (100.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 0 0k 0k ( 0.00%) ( 0.00%) tape 1 : 0 0k 0k ( 0.00%) PRODUCTION-002 4 dumpers idle : 0 taper status: Writing amdbkp-01:/test taper qlen: 0 network free kps: 600 holding space : 92835015k ( 95.05%) chunker0 busy : 0:00:41 ( 97.37%) dumper0 busy : 0:00:41 ( 97.35%) 0 dumpers busy : 0:00:01 ( 2.65%) 0: 0:00:01 (100.00%) 1 dumper busy : 0:00:41 ( 97.35%) 0: 0:00:41 (100.00%) amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -la /vtapes-offsite/slot2/ total 4838132 drwxr-xr-x 2 amandabackup disk 4096 Jul 16 17:14 . drwxrwxr-x 12 amandabackup disk 4096 Jul 16 17:13 .. -rw------- 1 amandabackup disk 32768 Jul 16 17:14 00000.PRODUCTION-002 -rw------- 1 amandabackup disk 4954193920 Jul 16 17:22 00001.amdbkp-01._test.0 </pre> I wasn's sure if amanda process was gone and not writing anything at that stage, so I checked it: <pre> amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ps -ef | grep am root 11046 897 0 16:31 ? 00:00:00 sshd: alex [priv] alex 11301 11046 0 16:31 ? 00:00:00 sshd: alex@pts/1 alex 11302 11301 0 16:31 pts/1 00:00:00 -bash 63998 12375 11405 0 17:12 pts/1 00:00:00 su - amandabackup 63998 12598 12376 0 17:22 pts/1 00:00:00 grep am [1]+ Done nohup amdump production-vtape-RAIT </pre> ====Test restore==== ====Test encryption==== ===Amanda clients=== ====Firewall policies==== You need the following policies in the firewall to allow Amanda server to talk to client and client to talk back to server. Below is an example for U3 pre-prod. [[file:Amanda firewall rules IN.png]] [[file:Amanda firewall rules OUT.png]] If you want to setup iptables for the same, please, follow [http://wiki.zmanda.com/index.php/How_To:Set_Up_iptables_for_Amanda this article]. ====Linux==== =====Linux (CentOS)===== ======Client installation====== *NOTE* Linux server infra-mon-02 was used for this example deployment. Download the package from [http://www.zmanda.com/download-amanda.php Amanda site]. <pre> [root@infra-mon-02 ]# cd /tmp && wget -c -t0 http://www.zmanda.com/downloads/community/Amanda/3.3.5/Redhat_Enterprise_6.0/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm </pre> Install pre-requisite packages: <pre> [root@infra-mon-02 ]# yum install xinetd perl-JSON </pre> Install Amanda backup client <pre> [root@infra-mon-02 ]# rpm -Uhv /tmp/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm </pre> Add Amanda server line to client's <tt>/var/lib/amanda/.amandahosts</tt> file <pre> localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump </pre> Start <tt>xinetd</tt> and confirm that it's listening on Amanda client port 10080: <pre> [root@infra-mon-02 ]# /etc/init.d/xinetd start [root@infra-mon-02 ]# netstat -anp | grep 1008 tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 2647/xinetd </pre> =====Linux (Ubuntu)===== ======Client installation====== *NOTE* Linux server nas2ftp-01 was used for this example deployment Download the package from [http://www.zmanda.com/download-amanda.php Amanda site]. <pre> # cd /tmp && wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb </pre> Install pre-requisite packages <tt>gettext</tt> and <tt>xinetd</tt> before you attempt to install Amanda client <pre> nas2ftp-01 # apt-get update nas2ftp-01 # apt-get install get text xinetd </pre> Now, install Amanda client <pre> nas2ftp-01 # dpkg -i /tmp/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb </pre> Add Amanda server line to client's <tt>/var/lib/amanda/.amandahosts</tt> file <pre> localhost root amindexd amidxtaped localhost.localdomain root amindexd amidxtaped localhost amandabackup amdump localhost.localdomain amandabackup amdump amdbkp-01.production amandabackup amdump </pre> If you have iptables active on the client, add the below line to the rule set (usually, it's in <tt>/etc/iptables.active.rules</tt>, if the client was provisioned from Linux VM template) <pre> # # allow Amanda in -A INPUT -s 192.168.0.23 -j ACCEPT </pre> Re-load iptables rules <pre> nas2ftp-01 # iptables-restore < /etc/iptables.active.rules </pre> =====Backup configuration on backup server===== Add what you want to backup to <tt>disklist</tt> file on Amanda server (below, I added <tt>/var/spool/cron/crontabs</tt> directory from nas2ftp-01 to be backup) <pre> nas2ftp-01 /var/spool/cron/crontabs/root tar-client-encrypt </pre> Run check to make sure that everything is fine from Amanda server prospective (you can ignore warning about absence of <tt>.exclude.lst</tt> file, if you don't use it to exclude something from backup): <pre> amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amcheck prod-1week-vtape-RAIT Amanda Tape Server Host Check ----------------------------- Holding disk /amanda: 188 GB disk space available, using 188 GB slot {8,8}: volume 'PRODUCTION-1WEEK-008' Will write to volume 'PRODUCTION-1WEEK-008' in slot {8,8}. NOTE: skipping tape-writable test NOTE: host info dir /etc/amanda/prod-1week-vtape-RAIT/curinfo/nas2ftp-01 does not exist NOTE: it will be created on the next run. NOTE: index dir /etc/amanda/prod-1week-vtape-RAIT/index/nas2ftp-01 does not exist NOTE: it will be created on the next run. Server check took 0.192 seconds Amanda Backup Client Hosts Check -------------------------------- ERROR: nas2ftp-01: [Can't open exclude file /var/spool/cron/crontabs/.exclude.lst (No such file or directory)] Client check: 1 host checked in 2.099 seconds. 1 problem found. (brought to you by Amanda 3.3.5) </pre> Now, we can run a test backup <pre> amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amdump prod-1week-vtape-RAIT nas2ftp-01 & [1] 17270 amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amstatus prod-1week-vtape-RAIT Using /etc/amanda/prod-1week-vtape-RAIT/log/amdump From Wed Oct 8 14:57:18 IST 2014 nas2ftp-01:/var/spool/cron/crontabs 0 0g estimate done SUMMARY part real estimated size size partition : 1 estimated : 1 0g flush : 0 0g failed : 0 0g ( 0.00%) wait for dumping: 0 0g ( 0.00%) dumping to tape : 0 0g ( 0.00%) dumping : 0 0g 0g ( 0.00%) ( 0.00%) dumped : 0 0g 0g ( 0.00%) ( 0.00%) wait for writing: 0 0g 0g ( 0.00%) ( 0.00%) wait to flush : 0 0g 0g (100.00%) ( 0.00%) writing to tape : 0 0g 0g ( 0.00%) ( 0.00%) failed to tape : 0 0g 0g ( 0.00%) ( 0.00%) taped : 0 0g 0g ( 0.00%) ( 0.00%) 4 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 80000 holding space : 188g (100.00%) 0 dumpers busy : 0:00:00 (100.00%) </pre> Shortly after you will have email report to say that the backup was good: <pre> Hostname: amdbkp-01 Org : production. 1 week cycle. Config : prod-1week-vtape-RAIT Date : October 8, 2014 These dumps were to tape PRODUCTION-1WEEK-008. The next 3 tapes Amanda expects to use are: 3 new tapes. The next 3 new tapes already labelled are: PRODUCTION-1WEEK-009, PRODUCTION-1WEEK-010, PRODUCTION-1WEEK-011 STATISTICS: Total Full Incr. Level:# -------- -------- -------- -------- Estimate Time (hrs:min) 0:00 Run Time (hrs:min) 0:00 Dump Time (hrs:min) 0:00 0:00 0:00 Output Size (meg) 0.0 0.0 0.0 Original Size (meg) 0.0 0.0 0.0 Avg Compressed Size (%) 110.0 110.0 -- DLEs Dumped 1 1 0 Avg Dump Rate (k/s) 89.4 89.4 -- Tape Time (hrs:min) 0:00 0:00 0:00 Tape Size (meg) 0.0 0.0 0.0 Tape Used (%) 0.0 0.0 0.0 DLEs Taped 1 1 0 Parts Taped 1 1 0 Avg Tp Write Rate (k/s) 100.0 100.0 -- USAGE BY TAPE: Label Time Size % DLEs Parts PRODUCTION-1WEEK-008 0:00 0G 0.0 1 1 NOTES: planner: Adding new disk nas2ftp-01:/var/spool/cron/crontabs. taper: Slot {8,8} with label PRODUCTION-1WEEK-008 is usable taper: tape PRODUCTION-1WEEK-008 kb 10 fm 1 [OK] DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK L ORIG-GB OUT-GB COMP% MMM:SS KB/s MMM:SS KB/s ------------------------------------------- ---------------------- -------------- ------------- nas2ftp-01 /var/spool/cron/crontabs 0 0 0 -- 0:00 89.4 0:00 100.0 (brought to you by Amanda version 3.3.5) </pre> ====Windows client==== =====Client installation===== Ref: * [http://wiki.zmanda.com/index.php/Zmanda_Windows_Client Zmanda Windows Client] * [http://kb.zmanda.com/article.php?id=467 How to create and use amandabackup domain user] There is amandabackup user account in PRODUCTION Windows domain created for Amanda client. Install Amanda client binaries: [[file:Windows client installer Amanda.PNG]] Supply amandabackup user password when prompted [[file:Windows client Amanda password prompt.PNG]] Enter Amanda server name when prompted: [[file:Windows client Amanda server prompt.PNG]] Check the amandabackup user is a member of local Administrators group on the Windows server: [[file:Windows client Amanda server local admins.PNG]] =====Backup configuration on backup server===== On Amanda server register the new client in the configuration that it meant to be in (prod-selligent-1week-vtape-RAIT configuration in the example below): <pre> amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amaddclient --config prod-selligent-1week-vtape-RAIT --client selrep-01.production --diskdev "E:/SSRS/SelligentReport" --dumptype zwc-compress Logging to /var/log/amanda/amaddclient.20141009171134.debug /etc/amanda/prod-selligent-1week-vtape-RAIT/disklist updated updating /var/lib/amanda/.amandahosts on amdbkp-01.production Attempting to update /var/lib/amanda/.amandahosts on selrep-01.production ssh: connect to host selrep-01.production port 22: Connection refused WARNING: scp from selrep-01.production not successful. Check selrep-01.production :/var/lib/amanda/.amandahosts file. If entry 'amdbkp-01.production amandabackup' is not present, append the entry to the file manually. Creating amanda-client.conf for selrep-01.production Creating /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production ssh: connect to host selrep-01.production port 22: Connection refused WARNING: Cannot create /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production Please copy /var/lib/amanda/amanda-client.conf-selrep-01.production to selrep-01.production manually File /var/lib/amanda/example/xinetd.amandaclient contains the latest Amanda client daemon configuration. Please merge it to /etc/xinetd.d/amandaclient. </pre> Run client check to make sure everything is working fine: <pre> amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amcheck -c prod-selligent-1week-vtape-RAIT Amanda Backup Client Hosts Check -------------------------------- Client check: 1 host checked in 0.236 seconds. 0 problems found. (brought to you by Amanda 3.3.5) </pre> ===Backup profile creation=== Now we want to add new backup profile for new application. *NOTE* Below I took BI as an example of the new profile You will find new profile creation script in <tt>/etc/amanda</tt> directory - <tt>new_profile.sh</tt>. The script will: * create Amanda profile directory * create two configuration files - with and without RAIT defined * create holding disk area for the profile * create specified number of vTapes on both NAS units - onsite and offsite Source of the script: <pre> #!/bin/sh # # The script creates new profile for Amanda backup # based on template file /var/lib/amanda/template.d/amanda--rait.conf # # NOTE: you might need to change the following by hand: # * dumpcycle (default: 1 weeks) # * runspercycle (default: 7) # * tapecycle (default: 14 tapes) # * tapetype (default: VTAPE-100G) # PROFILE_NAME=$1 NUM_OF_VTAPES=$2 PROFILE_NAME_LOW=`echo $PROFILE_NAME | awk '{print tolower($1)}'` PROFILE_NAME_UP=`echo $PROFILE_NAME | awk '{print toupper($1)}'` TEMPLATE=/var/lib/amanda/template.d/amanda-.conf TEMPLATE_RAIT=/var/lib/amanda/template.d/amanda--rait.conf AMANDA_CFG_DIR=/etc/amanda AMANDA_HOLDING_DISK_DIR=/amanda AMANDA_PROFILE_DIR=prod-$PROFILE_NAME_LOW-1week-vtape-RAIT VTAPES_DIR=/vtapes-prod/RAIT/1week VTAPES_OFFSITE_DIR=/vtapes-prod-offsite/RAIT/1week WARNING="NOTE: you might need to change the following by hand in $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf:\n dumpcycle (default: 1 weeks)\n runspercycle (default: 7)\n tapecycle (default: 14 tapes)\n tapetype (default: VTAPE-100G)\n" DISKLIST_HDR=" #\n # The below configured based on the requirements I had at that moment\n #\n" if [ $# -lt 2 ] then echo "\nUsage: `basename $0` <new_profile_name> <number_of_vtapes>\n" exit 1 fi if [ -f $TEMPLATE ] then if [ -d $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR ] then echo "\nERROR: $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR already exists!\n" exit 1 fi echo "Creating profile directory $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR \n" mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR # configuration without RAIT sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait # configuration with RAIT sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE_RAIT > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait # until we finished creating vTapes, we need RAIT configuration to be active ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf for d in log curinfo index do mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/${d} done echo $DISKLIST_HDR > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/disklist HOLDING_DISK_AREA=`amgetconf $AMANDA_PROFILE_DIR holdingdisk:hd1:directory` echo "Creating holding disk area for the profile - $HOLDING_DISK_AREA\n" if [ -d $AMANDA_HOLDING_DISK_DIR ] then if [ ! -d $HOLDING_DISK_AREA ] then mkdir $HOLDING_DISK_AREA else echo "Directory $HOLDING_DISK_AREA already exists!" fi else echo "ERROR: Holding disk directory $AMANDA_HOLDING_DISK_DIR does not exist!" fi echo "Creating $NUM_OF_VTAPES vTapes for the profile" for i in `seq -f "%03g" 1 $NUM_OF_VTAPES` do SN=`echo ${i}|sed -e 's/0*\([1-9].*\)/\1/g'` mkdir -p $VTAPES_DIR/$PROFILE_NAME_UP/slot$SN $VTAPES_OFFSITE_DIR/$PROFILE_NAME_UP/slot$SN && amlabel $AMANDA_PROFILE_DIR PRODUCTION-$PROFILE_NAME_UP-1WEEK-$i slot $SN done # switching to configuration without RAIT # should upgrade DC - HQ connection before we can use RAIT for all applications # for now RAIT is not in use. rm $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf echo "Profile $PROFILE_NAME_UP created!\n\n" echo $WARNING fi </pre> Engage the script to create BI profile with 14 vTapes: <pre> amandabackup@amdbkp-01:/etc/amanda$ ./new_profile.sh BI 14 Creating profile directory /etc/amanda/prod-sql-1week-vtape-RAIT Creating holding disk area for the profile - /amanda/prod-sql-1week-vtape-RAIT Directory /amanda/prod-sql-1week-vtape-RAIT already exists! Creating 14 vTapes for the profile Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-001'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-002'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-003'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-004'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-005'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-006'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-007'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-008'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-009'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-010'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-011'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-012'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-013'... Checking label... Success! Reading label... Found an empty tape. Writing label 'PRODUCTION-SQL-1WEEK-014'... Checking label... Success! Profile BI created! NOTE: you might need to change the following by hand in /etc/amanda/prod-sql-1week-vtape-RAIT/amanda.conf: dumpcycle (default: 1 weeks) runspercycle (default: 7) tapecycle (default: 14 tapes) tapetype (default: VTAPE-100G) </pre> Newly created <tt>amanda.conf</tt> points to Datacenter NAS only and <tt>amanda.conf-rait</tt> has RAIT (Datacenter NAS + HQ NAS) defined. Here is the diff between the two templates : <pre> --- /var/lib/amanda/template.d/amanda-.conf 2014-10-20 16:43:59.794381389 +0100 +++ /var/lib/amanda/template.d/amanda--rait.conf 2014-10-20 16:45:09.847107263 +0100 @@ -16,9 +16,16 @@ bumpmult 4 etimeout 1800 runtapes 3 -tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP" -changerfile "vtapes-statefile-tc" +tpchanger "chg-rait:{nas-tc,nas-gqh}" tapetype VTAPE-100G +define changer nas-tc { + tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP" + changerfile "vtapes-statefile-tc" +} +define changer nas-gqh { + tpchanger "chg-disk:/vtapes-prod-offsite/RAIT/1week/PROFILE_NAME_UP" + changerfile "vtapes-statefile-gqh" +} maxdumpsize -1 labelstr "^PRODUCTION-PROFILE_NAME_UP-1WEEK-[0-9][0-9]*$" holdingdisk hd1 { </pre> *WARNING* we should upgrade Datacenter - HQ connection before we can use RAIT for all applications. For now RAIT is not in use. Add required backup targets to <tt>disklist</tt> file. <pre> # # The below configured based on the requirements # SQL-01 I:/OBI/catalog zwc-encrypt-compress-server SQL-01 I:/OBI/GlobalCache zwc-encrypt-compress-server SQL-01 I:/OBI/repository zwc-encrypt-compress-server SQL-01 I:/Sources zwc-encrypt-compress-server </pre> Check that configuration is correct and working (ignore "does not exist" messages for info and index): <pre> amandabackup@amdbkp-01:/etc/amanda$ amcheck prod-sql-1week-vtape-RAIT Amanda Tape Server Host Check ----------------------------- Holding disk /amanda/prod-sql-1week-vtape-RAIT: 148107264 KB disk space available, using 148004864 KB slot 1: volume 'PRODUCTION-SQL-1WEEK-001' Will write to volume 'PRODUCTION-SQL-1WEEK-001' in slot 1. NOTE: skipping tape-writable test NOTE: host info dir /etc/amanda/prod-sql-1week-vtape-RAIT/curinfo/SQL-01 does not exist NOTE: it will be created on the next run. NOTE: index dir /etc/amanda/prod-sql-1week-vtape-RAIT/index/SQL-01 does not exist NOTE: it will be created on the next run. Server check took 0.174 seconds Amanda Backup Client Hosts Check -------------------------------- Client check: 1 host checked in 0.301 seconds. 0 problems found. (brought to you by Amanda 3.3.5) </pre> Create distribution list in you directory [[file:Amanda BI distribution list.png]] Add members to the list. Run backup and watch results: <pre> amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ nohup /etc/amanda/amdump_and_report.sh prod-sql-1week-vtape-RAIT & amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ watch -n3 amstatus prod-sql-1week-vtape-RAIT Every 3.0s: amstatus prod-sql-1week-vtape-RAIT Mon Oct 20 17:07:14 2014 Using /etc/amanda/prod-sql-1week-vtape-RAIT/log/amdump From Mon Oct 20 17:06:56 IST 2014 SQL-01:"I:/OBI/GlobalCache" 0 1k finished (17:07:04) SQL-01:"I:/OBI/catalog" 0 6001k dumping 672k ( 11.20%) (17:07:09) SQL-01:"I:/OBI/repository" 0 2339k finished (17:07:07) SQL-01:"I:/Sources" 0 357273k waiting for dumping SUMMARY part real estimated size size partition : 4 estimated : 4 364485k flush : 0 0k failed : 0 0k ( 0.00%) wait for dumping: 1 357273k ( 98.02%) dumping to tape : 0 0k ( 0.00%) dumping : 1 672k 6001k ( 11.20%) ( 0.18%) dumped : 2 2340k 1211k (193.23%) ( 0.64%) wait for writing: 0 0k 0k ( 0.00%) ( 0.00%) wait to flush : 0 0k 0k (100.00%) ( 0.00%) writing to tape : 0 0k 0k ( 0.00%) ( 0.00%) failed to tape : 0 0k 0k ( 0.00%) ( 0.00%) taped : 2 2340k 1211k (193.23%) ( 0.64%) tape 1 : 2 2338k 2338k ( 0.00%) PRODUCTION-SQL-1WEEK-001 (2 chunks) 3 dumpers idle : 0 taper status: Idle taper qlen: 0 network free kps: 78976 holding space : 147998784k (100.00%) chunker0 busy : 0:00:08 ( 60.97%) dumper0 busy : 0:00:08 ( 60.80%) taper busy : 0:00:00 ( 0.35%) 0 dumpers busy : 0:00:05 ( 39.20%) 0: 0:00:03 ( 60.64%) 2: 0:00:02 ( 39.34%) 1 dumper busy : 0:00:08 ( 60.80%) 0: 0:00:05 ( 67.10%) 5: 0:00:02 ( 32.90%) </pre> Since everything is confirmed to be working, you can add that distribution list you created in directory to the "mailto" parameter in <tt>amanda.conf*</tt> files. <pre> amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ grep mailto amanda.conf* amanda.conf:mailto "backup-reports@corporate eng@corporate" amanda.conf-norait:mailto "backup-reports@corporate eng@corporate" amanda.conf-rait:mailto "backup-reports@corporate eng@corporate" </pre> ===Logging and monitoring=== ====Kibana/Logstash==== We had Kibana and Logstash deployed and you could see logs from every run of Amanda backup in Kibana. See example of [[Wrapper script to report Amanda backup status to Logstash/Kibana]] Parsing of JSON output from Amanda requires improvement. We had few issues to resolve there: * There is a known limitation of Logstash - it can't explode JSON arrays properly. It leads to "summary" part of the output not parsed properly. * Amounts sent by Amanda is in kilobytes, but Kibana works with bytes and doesn't allow to do mathematical operations with the numbers in web UI. There is an option of doing it in the input filter at logstash level, but it would be preferable not to alter the original data on the way into logstash. After some discussion we thought that the better way was to ship the stats into Graphite and use Dashboard to plot the graphs (it allows calculations in web UI) ====Dashboard==== We had an idea to export data into Graphite, so we could build dashboards for backups. Amanda server statistics were available via Amanda backup server dashboard.

Return to Backup with Amanda: tape, NAS, Amazon S3.