Devstack. After bringing laptop back from sleep mode instance is in 'SHUTOFF' status and volume is in 'error deleting' status.
From MyWiki
(Difference between revisions)
Admin (Talk | contribs)
(Created page with 'After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance bac…')
Newer edit →
(Created page with 'After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance bac…')
Newer edit →
Revision as of 20:52, 9 January 2015
After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance back up failed and deleting instance didn't delete volume behind it.
Solution was found here
Our instance called test-instance and volume behind it is volume id 876771b2-04e6-4e80-81ba-6102deb6fa26
[stack@devstack devstack]$ nova list
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| 6d7174e8-6f47-417f-bd10-b163e77e504c | test-instance | SHUTOFF | - | Shutdown | private=172.16.20.5 |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
[stack@devstack devstack]$ openstack volume list
+--------------------------------------+--------------+----------------+------+----------+
| ID | Display Name | Status | Size | Attached |
+--------------------------------------+--------------+----------------+------+----------+
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | | error | 1 | |
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | | error_deleting | 1 | |
+--------------------------------------+--------------+----------------+------+----------+
[stack@devstack devstack]$ nova show test-instance
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack |
| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2014-12-24T03:38:09.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-12-24T03:37:53Z |
| flavor | m1.tiny (1) |
| hostId | 47c2aed6766c945e2da09de617a556e11c33b00a841efe70ed4fdd56 |
| id | 6d7174e8-6f47-417f-bd10-b163e77e504c |
| image | Attempt to boot from volume - no image supplied |
| key_name | devstack-ssh |
| metadata | {} |
| name | test-instance |
| os-extended-volumes:volumes_attached | [{"id": "876771b2-04e6-4e80-81ba-6102deb6fa26"}] |
| private network | 172.16.20.5 |
| security_groups | default, ssh_icmp |
| status | SHUTOFF |
| tenant_id | 56f102d9b8704e78bc13d80c92463eb5 |
| updated | 2015-01-08T19:22:20Z |
| user_id | 20cf3d728c444f6880fb1d554b9067f5 |
+--------------------------------------+----------------------------------------------------------+
[stack@devstack devstack]$ openstack volume show 876771b2-04e6-4e80-81ba-6102deb6fa26
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'device': u'vda', u'server_id': u'6d7174e8-6f47-417f-bd10-b163e77e504c', u'id': u'876771b2-04e6-4e80-81ba-6102deb6fa26', u'host_name': None, u'volume_id': u'876771b2-04e6-4e80-81ba-6102deb6fa26'}] |
| availability_zone | nova |
| bootable | true |
| created_at | 2014-12-24T03:37:55.000000 |
| display_description | |
| display_name | |
| encrypted | False |
| id | 876771b2-04e6-4e80-81ba-6102deb6fa26 |
| os-vol-host-attr:host | devstack |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| project_id | 56f102d9b8704e78bc13d80c92463eb5 |
| properties | attached_mode='rw', readonly='False' |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| type | None |
| volume_image_metadata | {u'kernel_id': u'20ee87ad-94b4-4cab-9d58-5959b7f51412', u'container_format': u'ami', u'min_ram': u'0', u'ramdisk_id': u'9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf', u'disk_format': u'ami', u'image_name': u'cirros-0.3.2-x86_64-uec', u'image_id': u'efdf518c-c021-47f6-94b1-fe93b0fa3969', u'checksum': u'4eada48c2843d2a262c814ddc92ecf2c', u'min_disk': u'0', u'size': u'25165824'} |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Volume 853e0bcc-05cf-484d-859d-1bcbfa46a55e was left over from the similar situation - sleep mode resulted in an instance left in limbo. Attempt to delete it fails:
[stack@devstack devstack]$ openstack volume delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e ERROR: openstack Invalid volume: Volume status must be available or error, but current status is: error_deleting (HTTP 400) (Request-ID: req-6e1f56ce-3805-41a9-bb68-d16de9dc35f7)
Let's try to delete the SHUTOFF instance:
[stack@devstack devstack]$ nova delete test-instance Request to delete server test-instance has been accepted. [stack@devstack devstack]$ nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+
Instance is gone, but not its volume 876771b2-04e6-4e80-81ba-6102deb6fa26, which is now also in 'error_deleting' state:
[stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
Check that we don't have any snapshots before we attempt anything else:
[stack@devstack devstack]$ cinder snapshot-list +----+-----------+--------+--------------+------+ | ID | Volume ID | Status | Display Name | Size | +----+-----------+--------+--------------+------+ +----+-----------+--------+--------------+------+
Let's check now what iSCSI targets do we have:
[stack@devstack devstack]$ sudo tgt-admin -s
Target 1: iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
Account information:
ACL information:
ALL
So, we have only one target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and no target for volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e Trying to delete the target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and it works fine:
[stack@devstack devstack]$ sudo tgt-admin -delete iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ sudo tgt-admin -s
But cinder force-delete is not doing the magic:
[stack@devstack devstack]$ cinder force-delete 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
OK. Let's try to reset-state and see if it helps:
[stack@devstack devstack]$ cinder reset-state 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder reset-state 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | available | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
Still nothing. Back to the same 'error_deleting' status.
Time for the tough measures - delete the records in MySQL volumes table:
[stack@devstack devstack]$ mysql -u root cinder Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 248 Server version: 5.1.73-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select * from volumes; +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | created_at | updated_at | deleted_at | deleted | id | ec2_id | user_id | project_id | host | size | availability_zone | instance_uuid | mountpoint | attach_time | status | attach_status | scheduled_at | launched_at | terminated_at | display_name | display_description | provider_location | provider_auth | snapshot_id | volume_type_id | source_volid | bootable | attached_host | provider_geometry | _name_id | encryption_key_id | migration_status | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | 2014-12-23 20:13:28 | 2014-12-24 03:34:07 | NULL | 0 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-23 20:13:29 | 2014-12-23 20:13:31 | 2014-12-24 03:34:06 | | | 172.16.10.10:3260,1 iqn.2010-10.org.openstack:volume-853e0bcc-05cf-484d-859d-1bcbfa46a55e 1 | CHAP Rt5PCuZJFwahV76e74u2 S6e7A2tSGcwyKvdd4PG5 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | | 2014-12-24 03:37:55 | 2015-01-09 08:58:15 | NULL | 0 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-24 03:37:55 | 2014-12-24 03:37:57 | 2015-01-09 08:58:15 | | | 172.16.10.10:3260,2 iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 1 | CHAP 5mgeSdMVPxTCPMFhqc2R wv4NJM9tVLKhWpZwpaB2 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ 2 rows in set (0.00 sec)
Attempt to delete volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e fails due to foreign key constraint:
mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_glance_metadata`, CONSTRAINT `volume_glance_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))
Check what we have in volume_glance_metadata:
mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-23 20:13:31 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | container_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 3 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_ram | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 4 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 5 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | disk_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 6 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 7 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 8 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 9 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_disk | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 10 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | size | 25165824 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 20 rows in set (0.00 sec)
Clean volume_glance_metadata now and try to delete the volume from volumes table again:
mysql> delete from volume_glance_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 10 rows affected (0.00 sec) mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 10 rows in set (0.00 sec) mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))
Boom! We have more in volume_admin_metadata. Clean it too:
mysql> select * from volume_admin_metadata; +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | key | value | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | 2014-12-23 20:13:34 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | readonly | False | | 2014-12-23 20:13:34 | NULL | 2014-12-24 03:34:06 | 1 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | attached_mode | rw | | 2014-12-24 03:37:59 | NULL | NULL | 0 | 3 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | readonly | False | | 2014-12-24 03:38:00 | NULL | 2015-01-09 08:58:14 | 1 | 4 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | attached_mode | rw | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ 4 rows in set (0.00 sec) mysql> delete from volume_admin_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 2 rows affected (0.00 sec)
Now again back to cleaning volumes table and it works this time:
mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 1 row affected (0.00 sec)
Repeat the process for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26:
mysql> delete from volume_admin_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 2 rows affected (0.00 sec) mysql> delete from volume_glance_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 10 rows affected (0.01 sec) mysql> delete from volumes where id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 1 row affected (0.00 sec) mysql> \q Bye
And it's all clean now:
[stack@devstack devstack]$ cinder list +----+--------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +----+--------+--------------+------+-------------+----------+-------------+ +----+--------+--------------+------+-------------+----------+-------------+
