Devstack installation with VirtualBox
From MyWiki
NOTE: this is still work in progress!
Contents |
Environment description
Assuming the following systems as a goal (nice ASCII picture idea is borrowed from here).
We will run Devstack in all-in-one VM under VirtualBox on Mac.
+----------+
| Client |
|(Mac Host)|
+-----+----+
|
------+--------+-------------------+-----------
|
+--------+--------+
|VM |
+-----------------+
| All-in-one Node |
| Nova |
| Keystone | +--------------------+
| Glance |----|nested VM (instance)|
| Neutron | +--------------------+
| Cinder | | |
| Horizon | | m1.tiny |
+-----------------+ | |
+--------------------+
VM setup
Get CentOS 6.6 VM setup. Basic server configuration.
Networking
We will have two networks for the environment:
- 172.16.10.0/24 - externally facing, NAT’ed
- 172.16.20.0/24 - internal subnet for instances
Good description of what is what in local.conf file from networking prospective is given here
Virtualbox
Add NAT network:
Mac-host:alex $ VBoxManage natnetwork add -t nat-int-network -n "172.16.10.0/24" -e -h on
Add NAT rules to allow access from host Mac to SSH, VNC and Web UI in the guest VM. We will need those when Devstack is up and running in the guest VM, so we can get into it from Mac host:
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "ssh:tcp:[]:3022:[172.16.10.10]:22"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "horizon:tcp:[]:8080:[172.16.10.10]:80"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "keystone:tcp:[]:5000:[172.16.10.10]:5000"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "vnc:tcp:[]:6080:[172.16.10.10]:6080"
Mac-host:alex $ VBoxManage list natnets
NetworkName: nat-int-network
IP: 172.16.10.1
Network: 172.16.10.0/24
IPv6 Enabled: No
IPv6 Prefix:
DHCP Enabled: Yes
Enabled: Yes
Port-forwarding (ipv4)
horizon:tcp:[]:8080:[172.16.10.10]:80
keystone:tcp:[]:5000:[172.16.10.10]:5000
ssh:tcp:[]:3022:[172.16.10.10]:22
vnc:tcp:[]:6080:[172.16.10.10]:6080
loopback mappings (ipv4)
127.0.0.1=2
If you made a mistake/typo while creating NAT rule, it's easy to remove it (ref Network Address Translation Service (experimental)):
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p delete ssh
VM
Spec:
- 2 vCPU
- 4GB of RAM
- eth0 - NAT to talk with outside world, static IP 172.16.10.10/255.255.255.0, 172.16.10.0/24
[root@devstack ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static IPADDR=172.16.10.10 NETMASK=255.255.255.0 GATEWAY=172.16.10.1
Now when we check routing table, something interesting is in there - 169.254.0.0 routes:
[root@devstack ~]# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 172.16.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 172.16.10.1 0.0.0.0 UG 0 0 0 eth0
Hinted by this post removed those by adding ‘NOZEROCONF=yes’ and restarting networking:
[root@devstack ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=devstack
NOZEROCONF=yes
[root@devstack ~]# /etc/init.d/network restart
Shutting down interface eth0: [ OK ]
Shutting down interface eth1: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: Determining if ip address 172.16.10.10 is already in use for device eth0...
[ OK ]
[root@devstack ~]# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
172.16.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
0.0.0.0 172.16.10.1 0.0.0.0 UG 0 0 0 eth0
Misc additional configuration
After OS is installed, do yum upgrade
# yum upgrade # yum install git # git clone https://github.com/openstack-dev/devstack.git -b stable/icehouse /opt/devstack # cd /opt/devstack/ # tools/create-stack-user.sh # chown -R stack:stack /opt/devstack
Switch off SELinux and iptables:
# vi /etc/sysconfig/selinux # setenforce 0 # getenforce # service iptables stop
Upgrade all packages to the latest (just to be up to date):
# yum upgrade # sudo -i -u stack
Get and set devstack
Create /opt/devstack/local.conf file
[stack@devstack devstack] cd /opt/devstack [stack@devstack devstack] vi local.conf
Below configuration file is based on this example
[[local|localrc]] # Credentials ADMIN_PASSWORD=devstack MYSQL_PASSWORD=devstack RABBIT_PASSWORD=devstack SERVICE_PASSWORD=devstack SERVICE_TOKEN=token #Enable/Disable Services disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron enable_service tempest # Networking HOST_IP=172.16.10.10 FLOATING_RANGE=172.16.20.0/24 PUBLIC_NETWORK_GATEWAY=172.16.20.1 FLAT_INTERFACE=eth0 #Stable Branches NOVA_BRANCH=stable/icehouse CINDER_BRANCH=stable/icehouse GLANCE_BRANCH=stable/icehouse HORIZON_BRANCH=stable/icehouse KEYSTONE_BRANCH=stable/icehouse NEUTRON_BRANCH=stable/icehouse #NEUTRON CONFIG Q_USE_DEBUG_COMMAND=True #CINDER CONFIG # Make this one bigger, if you have more space available to play with. # VOLUME_BACKING_FILE_SIZE=9216M #GENERAL CONFIG API_RATE_LIMIT=False # Timeouts ACTIVE_TIMEOUT=180 ASSOCIATE_TIMEOUT=60 BOOT_TIMEOUT=120 SERVICE_TIMEOUT=120 TERMINATE_TIMEOUT=60 # Default instance flavour DEFAULT_INSTANCE_TYPE=m1.tiny # Output LOGFILE=/opt/stack/logs/stack.sh.log VERBOSE=True LOG_COLOR=False SCREEN_LOGDIR=/opt/stack/logs
Let’s stack it now
[stack@devstack devstack] ./stack.sh
You will see this error:
[stack@devstack devstack]$ ./stack.sh
Loaded plugins: fastestmirror
Using mysql database backend
Traceback (most recent call last):
File "/opt/devstack/tools/outfilter.py", line 24, in <module>
import argparse
ImportError: No module named argparse
To fix it, install python-setuptools and argparse as was hinted in this post
[stack@devstack devstack] sudo yum install python-setuptools -y [stack@devstack devstack] sudo easy_install argparse
It will eventually fail again with the below:
2014-11-25 11:22:20.797 | ERROR: openstack Unknown command ['project', 'create', 'admin'] 2014-11-25 11:22:20.856 | + ADMIN_TENANT= 2014-11-25 11:22:20.857 | ++ grep ' id ' 2014-11-25 11:22:20.857 | ++ get_field 2 2014-11-25 11:22:20.859 | ++ read data 2014-11-25 11:22:20.860 | ++ openstack user create admin --project '' --email admin@example.com --password devstack 2014-11-25 11:22:21.723 | ERROR: openstack Invalid command '--project' 2014-11-25 11:22:21.774 | + ADMIN_USER= 2014-11-25 11:22:21.776 | ++ get_field 2 2014-11-25 11:22:21.776 | ++ grep ' id ' 2014-11-25 11:22:21.778 | ++ read data 2014-11-25 11:22:21.778 | ++ openstack role create admin 2014-11-25 11:22:22.643 | ERROR: openstack Unknown command ['role', 'create', 'admin'] 2014-11-25 11:22:22.695 | + ADMIN_ROLE= 2014-11-25 11:22:22.696 | + openstack role add --project --user 2014-11-25 11:22:23.499 | ERROR: openstack Invalid command '--project' 2014-11-25 11:22:23.541 | + exit_trap 2014-11-25 11:22:23.541 | + local r=2 2014-11-25 11:22:23.541 | ++ jobs -p 2014-11-25 11:22:23.542 | + jobs= 2014-11-25 11:22:23.542 | + [[ -n '' ]] 2014-11-25 11:22:23.542 | + exit 2
Upgrade setuptools and try again. Hint came from here
[stack@devstack devstack] sudo pip install --upgrade requests setuptools [stack@devstack devstack] ./unstack.sh [stack@devstack devstack] ./stack.sh
Bingo!
Horizon is now available at http://172.16.10.10/ Keystone is serving at http://172.16.10.10:5000/v2.0/ Examples on using novaclient command line is in exercise.sh The default users are: admin and demo The password: devstack This is your host ip: 172.16.10.10
Now run some tests:
[stack@devstack devstack] ./exercise.sh <...skipped loads of output...> ********************************************************************* SUCCESS: End DevStack Exercise: /opt/devstack/exercises/volumes.sh ********************************************************************* ===================================================================== SKIP marconi SKIP sahara SKIP swift SKIP trove PASS aggregates PASS boot_from_volume PASS bundle PASS client-args PASS client-env PASS euca PASS floating_ips PASS horizon PASS neutron-adv-test PASS sec_groups PASS volumes =====================================================================
Failed tests
Initially there were few tests failing for me:
FAILED euca FAILED floating_ips FAILED neutron-adv-test FAILED volumes
VM’s were not spinning up quickly enough or were not getting deleted while it was listed in the output and I could see the said VM’s in ‘nova list’ right after the tests were finished and reporting failure.
This post has very good ideas on how to overcome those failures.
For example, change DEFAULT_INSTANCE_TYPE to something small. Below are available flavours. Given that we only have 9G to play with, m1.tiny looks like a reasonable choice.
[stack@devstack devstack]$ nova flavor-list +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True | | 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True | | 451 | m1.heat | 512 | 0 | 0 | | 1 | 1.0 | True | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | | 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
Also, change default exercise timeouts. More on timeouts here
“FAILED volumes” and “/opt/stack/data/stack-volumes-backing-file does not exist” problem
This post helped to see the obvious – lack of disk space.
I copied and pasted the below without much thinking (bad!) from ‘Openstack in Action’ book into my local.conf
#CINDER CONFIGVOLUME_BACKING_FILE_SIZE=102400M
Having 100G of space defined in local.conf (see the above snippet) for stack-volumes-backing-file is not going to work when there is only 11G available :-)
[root@devstack ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_devstack-lv_root
17938864 6008860 11012092 36% /
tmpfs 961212 0 961212 0% /dev/shm
/dev/sda1 487652 64309 397743 14% /boot
To fix this, what I did was reducing VOLUME_BACKING_FILE_SIZE down to 9G
#CINDER CONFIGVOLUME_BACKING_FILE_SIZE=9216M
But this hasn’t solved the issue after I did unstack.sh && clean.sh && stack.sh. After digging logs in /opt/stack/logs, I found mentioning of volume group stack-volumes. And here it was:
[root@devstack ~]# vgs VG #PV #LV #SN Attr VSize VFree stack-volumes 1 0 0 wz--n- 10.01g 10.01g vg_devstack 1 2 0 wz--n- 19.51g 0
This post sent me in the right direction.
[root@devstack ~]# vgchange -a n stack-volumes && vgremove stack-volumes 0 logical volume(s) in volume group "stack-volumes" now active Volume group "stack-volumes" successfully removed
Switched back to clean and re-stack:
[stack@devstack devstack]$ ./unstack.sh && ./clean.sh && ./stack.sh
And here comes the missing file:
[root@devstack ~]# ls -al /opt/stack/data/ total 36 drwxr-xr-x. 7 stack root 4096 Nov 25 21:56 . drwxr-xr-x. 21 stack stack 4096 Nov 25 21:52 .. drwxr-xr-x. 3 stack stack 4096 Nov 25 21:53 cinder drwxr-xr-x. 4 stack stack 4096 Nov 25 21:53 glance drwxr-xr-x. 5 stack stack 4096 Nov 25 21:54 neutron drwxr-xr-x. 6 stack root 4096 Nov 25 21:55 nova -rw-r--r--. 1 stack stack 9663676416 Nov 25 21:53 stack-volumes-backing-file drwxr-xr-x. 2 stack stack 4096 Nov 25 21:56 tempest
CLI environment variables.
If you try to engage CLI at this stage, this is what you get:
[stack@devstack devstack]$ glance image-list You must provide a username via either --os-username or env[OS_USERNAME]
We need to setup environment variables to be able to use CLI.
If you scroll up after running the test script, you will see something like this:
++ OS_TENANT_NAME=demo ++ export OS_USERNAME=admin ++ OS_USERNAME=admin ++ export OS_PASSWORD=devstack ++ OS_PASSWORD=devstack ++ export OS_NO_CACHE=1 ++ OS_NO_CACHE=1 ++ HOST_IP=172.16.10.10 ++ SERVICE_HOST=172.16.10.10 ++ SERVICE_PROTOCOL=http ++ KEYSTONE_AUTH_PROTOCOL=http ++ GLANCE_HOST=172.16.10.10 ++ export OS_IDENTITY_API_VERSION=2.0 ++ OS_IDENTITY_API_VERSION=2.0 ++ export OS_AUTH_URL=http://172.16.10.10:5000/v2.0 ++ OS_AUTH_URL=http://172.16.10.10:5000/v2.0
Based on OpenStack docs we will need the below ones defined:
export OS_USERNAME=admin export OS_PASSWORD=devstack export OS_TENANT_NAME=demo export OS_AUTH_URL=http://172.16.10.10:5000/v2.0
Open ~stack/.bashrc and add those line to the bottom of the file; save & exit; re-read .bashrc and off you go:
[stack@devstack devstack]$ vi ~/.bashrc [stack@devstack devstack]$ . ~/.bashrc [stack@devstack devstack]$ glance image-list +--------------------------------------+--------------------------+-------------+------------------+-----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+--------------------------+-------------+------------------+-----------+--------+ | 5181ac8e-5f7b-4f61-a404-d30c4ff9db9f | cirros-0.3.2-x86_64-disk | qcow2 | bare | 13167616 | active | | 629e365d-09c5-46e8-bf6a-c0defb3672b2 | F17-x86_64-cfntools | qcow2 | bare | 476704768 | active | +--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
Reboot
I’m not yet sure if devstack is good to survive reboot of the VM it runs in, but the below seems to bring everything (as far as I can see for now) back up.
Make sure that Apache, RabbitMQ and MySQL are set to start on boot. It’s not set to start up on boot by stack.sh
[root@devstack ~]# chkconfig --level 235 httpd on [root@devstack ~]# chkconfig --level 235 mysqld on [root@devstack ~]# chkconfig --level 235 rabbitmq-server on
Don’t forget to setup loopback volume for the file-backed storage to be back available before run rejoin-stack.sh
[stack@devstack devstack]$ sudo losetup -f /opt/stack/data/stack-volumes-backing-file [stack@devstack devstack]$ ./rejoin-stack.sh
Cannot open your terminal ‘/dev/pts/0’ - please check.
After reboot you try to rejoin the stack and get this:
[stack@devstack devstack]$ ./rejoin-stack.sh Cannot open your terminal '/dev/pts/0' - please check. [stack@devstack ~]$ ls -l /dev/pts/0 crw--w----. 1 root tty 136, 0 Nov 26 12:49 /dev/pts/0
Just add read/write permission to the group and add stack user to tty group in /etc/group file:
[root@devstack ~]# chmod g+rw /dev/pts/0
Screens
Some useful hints in Linux screens in devstack
And useful commands from this post
[stack@devstack devstack]$ cd /opt/devstack [stack@devstack devstack]$ screen -dR ctrl-a :quit [stack@devstack devstack]$ ./rejoin-stack.sh ctrl-a d
