From MyWiki

(Difference between revisions)

Revision as of 21:00, 26 November 2014

NOTE: this is still work in progress!

1 Environment description
2 VM setup
- 2.1 Networking
  - 2.1.1 Virtualbox
  - 2.1.2 VM
- 2.2 Misc additional configuration
3 Get and set devstack
4 Failed tests
- 4.1 “FAILED volumes” and “/opt/stack/data/stack-volumes-backing-file does not exist” problem
5 CLI environment variables.
6 Reboot
- 6.1 Cannot open your terminal ‘/dev/pts/0’ - please check.
7 Screens

Environment description

Assuming the following systems as a goal (nice ASCII picture idea is borrowed from here).

We will run Devstack in all-in-one VM under VirtualBox on Mac.

+----------+
|  Client  |
|(Mac Host)|
+-----+----+
      |
------+--------+-------------------+-----------
               |         
      +--------+--------+
      |VM               |
      +-----------------+
      | All-in-one Node |
      |  Nova           |
      |  Keystone       |    +--------------------+
      |  Glance         |----|nested VM (instance)|
      |  Neutron        |    +--------------------+
      |  Cinder         |    |                    |
      |  Horizon        |    | m1.tiny            |
      +-----------------+    |                    |
                             +--------------------+

VM setup

Get CentOS 6.6 VM setup. Basic server configuration.

Networking

We will have two networks for the environment: * 172.16.10.0/24 - externally facing, NAT’ed * 172.16.20.0/24 - internal subnet for instances

Good description of what is what in local.conf file from networking prospective is given here

Virtualbox

Add NAT network:

Mac-host:alex $ VBoxManage natnetwork add -t nat-int-network -n "172.16.10.0/24" -e -h on

Add NAT rule to allow access to SSH and Web UI when Devstack is up and running into the guest VM that will be running Devstack, so we can get into it from Mac host command line:

Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "ssh:tcp:[]:3022:[172.16.10.10]:22"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "horizon:tcp:[]:8080:[172.16.10.10]:80"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "keystone:tcp:[]:5000:[172.16.10.10]:5000"
Mac-host:alex $ VBoxManage natnetwork modify -t nat-int-network -p "vnc:tcp:[]:6080:[172.16.10.10]:6080"

Mac-host:alex $ VBoxManage list natnets
NetworkName:    nat-int-network
IP:             172.16.10.1
Network:        172.16.10.0/24
IPv6 Enabled:   No
IPv6 Prefix:    
DHCP Enabled:   Yes
Enabled:        Yes
Port-forwarding (ipv4)
        horizon:tcp:[]:8080:[172.16.10.10]:80
        keystone:tcp:[]:5000:[172.16.10.10]:5000
        ssh:tcp:[]:3022:[172.16.10.10]:22
        vnc:tcp:[]:6080:[172.16.10.10]:6080
loopback mappings (ipv4)
        127.0.0.1=2

VM

2 CPUs, 4GB of RAM, eth0 eth0 - NAT to talk with outside world, static IP 172.16.10.10/255.255.255.0, 172.16.10.0/24

[root@devstack ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=172.16.10.10
NETMASK=255.255.255.0
GATEWAY=172.16.10.1

Now when we check routing table, something interesting is in there - 169.254.0.0 routes:

[root@devstack ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
172.16.10.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         172.16.10.1     0.0.0.0         UG        0 0          0 eth0

Hinted by this post removed those by adding ‘NOZEROCONF=yes’ and restarting networking:

[root@devstack ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=devstack
NOZEROCONF=yes

[root@devstack ~]# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  Determining if ip address 172.16.10.10 is already in use for device eth0...
                                                           [  OK  ]
[root@devstack ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
172.16.10.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0
0.0.0.0         172.16.10.1     0.0.0.0         UG        0 0          0 eth0

Misc additional configuration

After OS is installed, do yum upgrade

# yum upgrade

# yum install git

# git clone https://github.com/openstack-dev/devstack.git -b stable/icehouse /opt/devstack

# cd /opt/devstack/
# tools/create-stack-user.sh 
# chown -R stack:stack /opt/devstack

Switch off SELinux and iptables:

# vi /etc/sysconfig/selinux 
# setenforce 0
# getenforce 
# service iptables stop

Upgrade all packages to the latest (just to be up to date):

# yum upgrade
# sudo -i -u stack

Get and set devstack

Create /opt/devstack/local.conf file

[stack@devstack devstack] cd /opt/devstack
[stack@devstack devstack] vi local.conf

Below configuration file is based on this example

[[local|localrc]]
# Credentials
ADMIN_PASSWORD=devstack
MYSQL_PASSWORD=devstack
RABBIT_PASSWORD=devstack
SERVICE_PASSWORD=devstack
SERVICE_TOKEN=token

#Enable/Disable Services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service tempest

# Networking
HOST_IP=172.16.10.10
FLOATING_RANGE=172.16.20.0/24
PUBLIC_NETWORK_GATEWAY=172.16.20.1
FLAT_INTERFACE=eth0

#Stable Branches
NOVA_BRANCH=stable/icehouse
CINDER_BRANCH=stable/icehouse
GLANCE_BRANCH=stable/icehouse
HORIZON_BRANCH=stable/icehouse
KEYSTONE_BRANCH=stable/icehouse
NEUTRON_BRANCH=stable/icehouse

#NEUTRON CONFIG
Q_USE_DEBUG_COMMAND=True

#CINDER CONFIG
# Make this one bigger, if you have more space available to play with.
#
VOLUME_BACKING_FILE_SIZE=9216M

#GENERAL CONFIG
API_RATE_LIMIT=False

# Timeouts
ACTIVE_TIMEOUT=180
ASSOCIATE_TIMEOUT=60
BOOT_TIMEOUT=120
SERVICE_TIMEOUT=120
TERMINATE_TIMEOUT=60

# Default instance flavour
DEFAULT_INSTANCE_TYPE=m1.tiny

# Output
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs

Let’s stack it now

[stack@devstack devstack] ./stack.sh

You will see this error:

[stack@devstack devstack]$ ./stack.sh 
Loaded plugins: fastestmirror
Using mysql database backend
Traceback (most recent call last):
  File "/opt/devstack/tools/outfilter.py", line 24, in <module>
    import argparse
ImportError: No module named argparse

To fix it, install python-setuptools and argparse as was hinted in this post

[stack@devstack devstack] sudo yum install python-setuptools -y
[stack@devstack devstack] sudo easy_install argparse

It will eventually fail again with the below:

2014-11-25 11:22:20.797 | ERROR: openstack Unknown command ['project', 'create', 'admin']
2014-11-25 11:22:20.856 | + ADMIN_TENANT=
2014-11-25 11:22:20.857 | ++ grep ' id '
2014-11-25 11:22:20.857 | ++ get_field 2
2014-11-25 11:22:20.859 | ++ read data
2014-11-25 11:22:20.860 | ++ openstack user create admin --project '' --email admin@example.com --password devstack
2014-11-25 11:22:21.723 | ERROR: openstack Invalid command '--project'
2014-11-25 11:22:21.774 | + ADMIN_USER=
2014-11-25 11:22:21.776 | ++ get_field 2
2014-11-25 11:22:21.776 | ++ grep ' id '
2014-11-25 11:22:21.778 | ++ read data
2014-11-25 11:22:21.778 | ++ openstack role create admin
2014-11-25 11:22:22.643 | ERROR: openstack Unknown command ['role', 'create', 'admin']
2014-11-25 11:22:22.695 | + ADMIN_ROLE=
2014-11-25 11:22:22.696 | + openstack role add --project --user
2014-11-25 11:22:23.499 | ERROR: openstack Invalid command '--project'
2014-11-25 11:22:23.541 | + exit_trap
2014-11-25 11:22:23.541 | + local r=2
2014-11-25 11:22:23.541 | ++ jobs -p
2014-11-25 11:22:23.542 | + jobs=
2014-11-25 11:22:23.542 | + [[ -n '' ]]
2014-11-25 11:22:23.542 | + exit 2

Upgrade setuptools and try again. Hint came from here

[stack@devstack devstack] sudo pip install --upgrade requests setuptools
[stack@devstack devstack] ./unstack.sh
[stack@devstack devstack] ./stack.sh

Bingo!

Horizon is now available at http://172.16.10.10/
Keystone is serving at http://172.16.10.10:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: devstack
This is your host ip: 172.16.10.10

Now run some tests:

[stack@devstack devstack] ./exercise.sh 

<...skipped loads of output...>

*********************************************************************
SUCCESS: End DevStack Exercise: /opt/devstack/exercises/volumes.sh
*********************************************************************
=====================================================================
SKIP marconi
SKIP sahara
SKIP swift
SKIP trove
PASS aggregates
PASS boot_from_volume
PASS bundle
PASS client-args
PASS client-env
PASS euca
PASS floating_ips
PASS horizon
PASS neutron-adv-test
PASS sec_groups
PASS volumes
=====================================================================

Failed tests

Initially there were few tests failing for me:

FAILED euca
FAILED floating_ips
FAILED neutron-adv-test
FAILED volumes

VM’s were not spinning up quickly enough or were not getting deleted while it was listed in the output and I could see the said VM’s in ‘nova list’ right after the tests were finished and reporting failure.

This post has very good ideas on how to overcome those failures.

For example, change DEFAULT_INSTANCE_TYPE to something small. Below are available flavours. Given that we only have 9G to play with, m1.tiny looks like a reasonable choice.

[stack@devstack devstack]$ nova flavor-list
+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID  | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1   | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2   | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3   | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4   | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 42  | m1.nano   | 64        | 0    | 0         |      | 1     | 1.0         | True      |
| 451 | m1.heat   | 512       | 0    | 0         |      | 1     | 1.0         | True      |
| 5   | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
| 84  | m1.micro  | 128       | 0    | 0         |      | 1     | 1.0         | True      |
+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

Also, change default exercise timeouts. More on timeouts here

“FAILED volumes” and “/opt/stack/data/stack-volumes-backing-file does not exist” problem

This post helped to see the obvious – lack of disk space.

I copied and pasted the below without much thinking (bad!) from ‘Openstack in Action’ book into my local.conf

#CINDER CONFIGVOLUME_BACKING_FILE_SIZE=102400M

Having 100G of space defined in local.conf (see the above snippet) for stack-volumes-backing-file is not going to work when there is only 11G available :-)

[root@devstack ~]# df -k
Filesystem           1K-blocks    Used Available Use% Mounted on
/dev/mapper/vg_devstack-lv_root
                      17938864 6008860  11012092  36% /
tmpfs                   961212       0    961212   0% /dev/shm
/dev/sda1               487652   64309    397743  14% /boot

To fix this, what I did was reducing VOLUME_BACKING_FILE_SIZE down to 9G

#CINDER CONFIGVOLUME_BACKING_FILE_SIZE=9216M

But this hasn’t solved the issue after I did unstack.sh && clean.sh && stack.sh. After digging logs in /opt/stack/logs, I found mentioning of volume group stack-volumes. And here it was:

[root@devstack ~]# vgs
  VG            #PV #LV #SN Attr   VSize  VFree 
  stack-volumes   1   0   0 wz--n- 10.01g 10.01g
  vg_devstack     1   2   0 wz--n- 19.51g     0

This post sent me in the right direction.

[root@devstack ~]# vgchange -a n stack-volumes && vgremove stack-volumes
  0 logical volume(s) in volume group "stack-volumes" now active
  Volume group "stack-volumes" successfully removed

Switched back to clean and re-stack:

[stack@devstack devstack]$ ./unstack.sh && ./clean.sh && ./stack.sh

And here comes the missing file:

[root@devstack ~]# ls -al /opt/stack/data/
total 36
drwxr-xr-x.  7 stack root        4096 Nov 25 21:56 .
drwxr-xr-x. 21 stack stack       4096 Nov 25 21:52 ..
drwxr-xr-x.  3 stack stack       4096 Nov 25 21:53 cinder
drwxr-xr-x.  4 stack stack       4096 Nov 25 21:53 glance
drwxr-xr-x.  5 stack stack       4096 Nov 25 21:54 neutron
drwxr-xr-x.  6 stack root        4096 Nov 25 21:55 nova
-rw-r--r--.  1 stack stack 9663676416 Nov 25 21:53 stack-volumes-backing-file
drwxr-xr-x.  2 stack stack       4096 Nov 25 21:56 tempest

CLI environment variables.

If you try to engage CLI at this stage, this is what you get:

[stack@devstack devstack]$ glance image-list
You must provide a username via either --os-username or env[OS_USERNAME]

We need to setup environment variables to be able to use CLI.

If you scroll up after running the test script, you will see something like this:

++ OS_TENANT_NAME=demo
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export OS_PASSWORD=devstack
++ OS_PASSWORD=devstack
++ export OS_NO_CACHE=1
++ OS_NO_CACHE=1
++ HOST_IP=172.16.10.10
++ SERVICE_HOST=172.16.10.10
++ SERVICE_PROTOCOL=http
++ KEYSTONE_AUTH_PROTOCOL=http
++ GLANCE_HOST=172.16.10.10
++ export OS_IDENTITY_API_VERSION=2.0
++ OS_IDENTITY_API_VERSION=2.0
++ export OS_AUTH_URL=http://172.16.10.10:5000/v2.0
++ OS_AUTH_URL=http://172.16.10.10:5000/v2.0

Based on OpenStack docs we will need the below ones defined:

export OS_USERNAME=admin
export OS_PASSWORD=devstack
export OS_TENANT_NAME=demo
export OS_AUTH_URL=http://172.16.10.10:5000/v2.0

Open ~stack/.bashrc and add those line to the bottom of the file; save & exit; re-read .bashrc and off you go:

[stack@devstack devstack]$ vi ~/.bashrc 
[stack@devstack devstack]$ . ~/.bashrc 
[stack@devstack devstack]$ glance image-list

+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                     | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| 5181ac8e-5f7b-4f61-a404-d30c4ff9db9f | cirros-0.3.2-x86_64-disk | qcow2       | bare             | 13167616  | active |
| 629e365d-09c5-46e8-bf6a-c0defb3672b2 | F17-x86_64-cfntools      | qcow2       | bare             | 476704768 | active |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+

Reboot

I’m not yet sure if devstack is good to survive reboot of the VM it runs in, but the below seems to bring everything (as far as I can see for now) back up.

Make sure that Apache, RabbitMQ and MySQL are set to start on boot. It’s not set to start up on boot by stack.sh

[root@devstack ~]# chkconfig --level 235 httpd on
[root@devstack ~]# chkconfig --level 235 mysqld on
[root@devstack ~]# chkconfig --level 235 rabbitmq-server on

Don’t forget to setup loopback volume for the file-backed storage to be back available before run rejoin-stack.sh

[stack@devstack devstack]$ sudo losetup -f /opt/stack/data/stack-volumes-backing-file
[stack@devstack devstack]$ ./rejoin-stack.sh

Cannot open your terminal ‘/dev/pts/0’ - please check.

After reboot you try to rejoin the stack and get this:

[stack@devstack devstack]$ ./rejoin-stack.sh 
Cannot open your terminal '/dev/pts/0' - please check.

[stack@devstack ~]$ ls -l /dev/pts/0
crw--w----. 1 root tty 136, 0 Nov 26 12:49 /dev/pts/0

Just add read/write permission to the group and add stack user to tty group in /etc/group file:

[root@devstack ~]# chmod g+rw /dev/pts/0

Screens

Some useful hints in Linux screens in devstack

And useful commands from this post

[stack@devstack devstack]$ cd /opt/devstack 
[stack@devstack devstack]$ screen -dR 
ctrl-a 
:quit 

[stack@devstack devstack]$ ./rejoin-stack.sh 
ctrl-a 
d

Devstack installation with VirtualBox