Devstack. After bringing laptop back from sleep mode instance is in 'SHUTOFF' status and volume is in 'error deleting' status.
From MyWiki
(Difference between revisions)
(Created page with 'After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance bac…') |
m (Protected "Devstack. After bringing laptop back from sleep mode instance is in 'SHUTOFF' status and volume is in 'error deleting' status." ([edit=sysop] (indefinite) [move=sysop] (indefinite))) |
Current revision as of 20:52, 9 January 2015
After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance back up failed and deleting instance didn't delete volume behind it.
Solution was found here
Our instance called test-instance and volume behind it is volume id 876771b2-04e6-4e80-81ba-6102deb6fa26
[stack@devstack devstack]$ nova list
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| 6d7174e8-6f47-417f-bd10-b163e77e504c | test-instance | SHUTOFF | - | Shutdown | private=172.16.20.5 |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
[stack@devstack devstack]$ openstack volume list
+--------------------------------------+--------------+----------------+------+----------+
| ID | Display Name | Status | Size | Attached |
+--------------------------------------+--------------+----------------+------+----------+
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | | error | 1 | |
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | | error_deleting | 1 | |
+--------------------------------------+--------------+----------------+------+----------+
[stack@devstack devstack]$ nova show test-instance
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack |
| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2014-12-24T03:38:09.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-12-24T03:37:53Z |
| flavor | m1.tiny (1) |
| hostId | 47c2aed6766c945e2da09de617a556e11c33b00a841efe70ed4fdd56 |
| id | 6d7174e8-6f47-417f-bd10-b163e77e504c |
| image | Attempt to boot from volume - no image supplied |
| key_name | devstack-ssh |
| metadata | {} |
| name | test-instance |
| os-extended-volumes:volumes_attached | [{"id": "876771b2-04e6-4e80-81ba-6102deb6fa26"}] |
| private network | 172.16.20.5 |
| security_groups | default, ssh_icmp |
| status | SHUTOFF |
| tenant_id | 56f102d9b8704e78bc13d80c92463eb5 |
| updated | 2015-01-08T19:22:20Z |
| user_id | 20cf3d728c444f6880fb1d554b9067f5 |
+--------------------------------------+----------------------------------------------------------+
[stack@devstack devstack]$ openstack volume show 876771b2-04e6-4e80-81ba-6102deb6fa26
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'device': u'vda', u'server_id': u'6d7174e8-6f47-417f-bd10-b163e77e504c', u'id': u'876771b2-04e6-4e80-81ba-6102deb6fa26', u'host_name': None, u'volume_id': u'876771b2-04e6-4e80-81ba-6102deb6fa26'}] |
| availability_zone | nova |
| bootable | true |
| created_at | 2014-12-24T03:37:55.000000 |
| display_description | |
| display_name | |
| encrypted | False |
| id | 876771b2-04e6-4e80-81ba-6102deb6fa26 |
| os-vol-host-attr:host | devstack |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| project_id | 56f102d9b8704e78bc13d80c92463eb5 |
| properties | attached_mode='rw', readonly='False' |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| type | None |
| volume_image_metadata | {u'kernel_id': u'20ee87ad-94b4-4cab-9d58-5959b7f51412', u'container_format': u'ami', u'min_ram': u'0', u'ramdisk_id': u'9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf', u'disk_format': u'ami', u'image_name': u'cirros-0.3.2-x86_64-uec', u'image_id': u'efdf518c-c021-47f6-94b1-fe93b0fa3969', u'checksum': u'4eada48c2843d2a262c814ddc92ecf2c', u'min_disk': u'0', u'size': u'25165824'} |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Volume 853e0bcc-05cf-484d-859d-1bcbfa46a55e was left over from the similar situation - sleep mode resulted in an instance left in limbo. Attempt to delete it fails:
[stack@devstack devstack]$ openstack volume delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e ERROR: openstack Invalid volume: Volume status must be available or error, but current status is: error_deleting (HTTP 400) (Request-ID: req-6e1f56ce-3805-41a9-bb68-d16de9dc35f7)
Let's try to delete the SHUTOFF instance:
[stack@devstack devstack]$ nova delete test-instance Request to delete server test-instance has been accepted. [stack@devstack devstack]$ nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+
Instance is gone, but not its volume 876771b2-04e6-4e80-81ba-6102deb6fa26, which is now also in 'error_deleting' state:
[stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
Check that we don't have any snapshots before we attempt anything else:
[stack@devstack devstack]$ cinder snapshot-list +----+-----------+--------+--------------+------+ | ID | Volume ID | Status | Display Name | Size | +----+-----------+--------+--------------+------+ +----+-----------+--------+--------------+------+
Let's check now what iSCSI targets do we have:
[stack@devstack devstack]$ sudo tgt-admin -s
Target 1: iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
Account information:
ACL information:
ALL
So, we have only one target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and no target for volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e Trying to delete the target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and it works fine:
[stack@devstack devstack]$ sudo tgt-admin -delete iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ sudo tgt-admin -s
But cinder force-delete is not doing the magic:
[stack@devstack devstack]$ cinder force-delete 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
OK. Let's try to reset-state and see if it helps:
[stack@devstack devstack]$ cinder reset-state 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder reset-state 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | available | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
Still nothing. Back to the same 'error_deleting' status.
Time for the tough measures - delete the records in MySQL volumes table:
[stack@devstack devstack]$ mysql -u root cinder Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 248 Server version: 5.1.73-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select * from volumes; +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | created_at | updated_at | deleted_at | deleted | id | ec2_id | user_id | project_id | host | size | availability_zone | instance_uuid | mountpoint | attach_time | status | attach_status | scheduled_at | launched_at | terminated_at | display_name | display_description | provider_location | provider_auth | snapshot_id | volume_type_id | source_volid | bootable | attached_host | provider_geometry | _name_id | encryption_key_id | migration_status | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | 2014-12-23 20:13:28 | 2014-12-24 03:34:07 | NULL | 0 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-23 20:13:29 | 2014-12-23 20:13:31 | 2014-12-24 03:34:06 | | | 172.16.10.10:3260,1 iqn.2010-10.org.openstack:volume-853e0bcc-05cf-484d-859d-1bcbfa46a55e 1 | CHAP Rt5PCuZJFwahV76e74u2 S6e7A2tSGcwyKvdd4PG5 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | | 2014-12-24 03:37:55 | 2015-01-09 08:58:15 | NULL | 0 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-24 03:37:55 | 2014-12-24 03:37:57 | 2015-01-09 08:58:15 | | | 172.16.10.10:3260,2 iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 1 | CHAP 5mgeSdMVPxTCPMFhqc2R wv4NJM9tVLKhWpZwpaB2 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ 2 rows in set (0.00 sec)
Attempt to delete volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e fails due to foreign key constraint:
mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_glance_metadata`, CONSTRAINT `volume_glance_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))
Check what we have in volume_glance_metadata:
mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-23 20:13:31 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | container_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 3 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_ram | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 4 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 5 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | disk_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 6 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 7 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 8 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 9 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_disk | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 10 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | size | 25165824 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 20 rows in set (0.00 sec)
Clean volume_glance_metadata now and try to delete the volume from volumes table again:
mysql> delete from volume_glance_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 10 rows affected (0.00 sec) mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 10 rows in set (0.00 sec) mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))
Boom! We have more in volume_admin_metadata. Clean it too:
mysql> select * from volume_admin_metadata; +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | key | value | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | 2014-12-23 20:13:34 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | readonly | False | | 2014-12-23 20:13:34 | NULL | 2014-12-24 03:34:06 | 1 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | attached_mode | rw | | 2014-12-24 03:37:59 | NULL | NULL | 0 | 3 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | readonly | False | | 2014-12-24 03:38:00 | NULL | 2015-01-09 08:58:14 | 1 | 4 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | attached_mode | rw | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ 4 rows in set (0.00 sec) mysql> delete from volume_admin_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 2 rows affected (0.00 sec)
Now again back to cleaning volumes table and it works this time:
mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 1 row affected (0.00 sec)
Repeat the process for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26:
mysql> delete from volume_admin_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 2 rows affected (0.00 sec) mysql> delete from volume_glance_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 10 rows affected (0.01 sec) mysql> delete from volumes where id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 1 row affected (0.00 sec) mysql> \q Bye
And it's all clean now:
[stack@devstack devstack]$ cinder list +----+--------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +----+--------+--------------+------+-------------+----------+-------------+ +----+--------+--------------+------+-------------+----------+-------------+
