View source

From MyWiki

You do not have permission to edit this page, for the following reasons:

• The action you have requested is limited to users in the group: Administrators.
• This page has been locked to prevent editing.

You can view and copy the source of this page:

After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance back up failed and deleting instance didn't delete volume behind it. Solution was found [https://ask.openstack.org/en/question/1378/cannot-remove-volume/ here] Our instance called test-instance and volume behind it is volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 <pre> [stack@devstack devstack]$ nova list +--------------------------------------+---------------+---------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------------+---------+------------+-------------+---------------------+ | 6d7174e8-6f47-417f-bd10-b163e77e504c | test-instance | SHUTOFF | - | Shutdown | private=172.16.20.5 | +--------------------------------------+---------------+---------+------------+-------------+---------------------+ [stack@devstack devstack]$ openstack volume list +--------------------------------------+--------------+----------------+------+----------+ | ID | Display Name | Status | Size | Attached | +--------------------------------------+--------------+----------------+------+----------+ | 876771b2-04e6-4e80-81ba-6102deb6fa26 | | error | 1 | | | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | | error_deleting | 1 | | +--------------------------------------+--------------+----------------+------+----------+ [stack@devstack devstack]$ nova show test-instance +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | AUTO | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | devstack | | OS-EXT-SRV-ATTR:hypervisor_hostname | devstack | | OS-EXT-SRV-ATTR:instance_name | instance-00000002 | | OS-EXT-STS:power_state | 4 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | stopped | | OS-SRV-USG:launched_at | 2014-12-24T03:38:09.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2014-12-24T03:37:53Z | | flavor | m1.tiny (1) | | hostId | 47c2aed6766c945e2da09de617a556e11c33b00a841efe70ed4fdd56 | | id | 6d7174e8-6f47-417f-bd10-b163e77e504c | | image | Attempt to boot from volume - no image supplied | | key_name | devstack-ssh | | metadata | {} | | name | test-instance | | os-extended-volumes:volumes_attached | [{"id": "876771b2-04e6-4e80-81ba-6102deb6fa26"}] | | private network | 172.16.20.5 | | security_groups | default, ssh_icmp | | status | SHUTOFF | | tenant_id | 56f102d9b8704e78bc13d80c92463eb5 | | updated | 2015-01-08T19:22:20Z | | user_id | 20cf3d728c444f6880fb1d554b9067f5 | +--------------------------------------+----------------------------------------------------------+ [stack@devstack devstack]$ openstack volume show 876771b2-04e6-4e80-81ba-6102deb6fa26 +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{u'device': u'vda', u'server_id': u'6d7174e8-6f47-417f-bd10-b163e77e504c', u'id': u'876771b2-04e6-4e80-81ba-6102deb6fa26', u'host_name': None, u'volume_id': u'876771b2-04e6-4e80-81ba-6102deb6fa26'}] | | availability_zone | nova | | bootable | true | | created_at | 2014-12-24T03:37:55.000000 | | display_description | | | display_name | | | encrypted | False | | id | 876771b2-04e6-4e80-81ba-6102deb6fa26 | | os-vol-host-attr:host | devstack | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | project_id | 56f102d9b8704e78bc13d80c92463eb5 | | properties | attached_mode='rw', readonly='False' | | size | 1 | | snapshot_id | None | | source_volid | None | | status | error | | type | None | | volume_image_metadata | {u'kernel_id': u'20ee87ad-94b4-4cab-9d58-5959b7f51412', u'container_format': u'ami', u'min_ram': u'0', u'ramdisk_id': u'9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf', u'disk_format': u'ami', u'image_name': u'cirros-0.3.2-x86_64-uec', u'image_id': u'efdf518c-c021-47f6-94b1-fe93b0fa3969', u'checksum': u'4eada48c2843d2a262c814ddc92ecf2c', u'min_disk': u'0', u'size': u'25165824'} | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ </pre> Volume 853e0bcc-05cf-484d-859d-1bcbfa46a55e was left over from the similar situation - sleep mode resulted in an instance left in limbo. Attempt to delete it fails: <pre> [stack@devstack devstack]$ openstack volume delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e ERROR: openstack Invalid volume: Volume status must be available or error, but current status is: error_deleting (HTTP 400) (Request-ID: req-6e1f56ce-3805-41a9-bb68-d16de9dc35f7) </pre> Let's try to delete the SHUTOFF instance: <pre> [stack@devstack devstack]$ nova delete test-instance Request to delete server test-instance has been accepted. [stack@devstack devstack]$ nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+ </pre> Instance is gone, but not its volume 876771b2-04e6-4e80-81ba-6102deb6fa26, which is now also in 'error_deleting' state: <pre> [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ </pre> Check that we don't have any snapshots before we attempt anything else: <pre> [stack@devstack devstack]$ cinder snapshot-list +----+-----------+--------+--------------+------+ | ID | Volume ID | Status | Display Name | Size | +----+-----------+--------+--------------+------+ +----+-----------+--------+--------------+------+ </pre> Let's check now what iSCSI targets do we have: <pre> [stack@devstack devstack]$ sudo tgt-admin -s Target 1: iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: Account information: ACL information: ALL </pre> So, we have only one target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and no target for volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e Trying to delete the target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and it works fine: <pre> [stack@devstack devstack]$ sudo tgt-admin -delete iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ sudo tgt-admin -s </pre> But cinder force-delete is not doing the magic: <pre> [stack@devstack devstack]$ cinder force-delete 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ </pre> OK. Let's try to reset-state and see if it helps: <pre> [stack@devstack devstack]$ cinder reset-state 876771b2-04e6-4e80-81ba-6102deb6fa26 [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder reset-state 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | available | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e [stack@devstack devstack]$ cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting | | 1 | None | true | | | 876771b2-04e6-4e80-81ba-6102deb6fa26 | available | | 1 | None | true | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ </pre> Still nothing. Back to the same 'error_deleting' status. Time for the tough measures - delete the records in MySQL volumes table: <pre> [stack@devstack devstack]$ mysql -u root cinder Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 248 Server version: 5.1.73-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select * from volumes; +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | created_at | updated_at | deleted_at | deleted | id | ec2_id | user_id | project_id | host | size | availability_zone | instance_uuid | mountpoint | attach_time | status | attach_status | scheduled_at | launched_at | terminated_at | display_name | display_description | provider_location | provider_auth | snapshot_id | volume_type_id | source_volid | bootable | attached_host | provider_geometry | _name_id | encryption_key_id | migration_status | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ | 2014-12-23 20:13:28 | 2014-12-24 03:34:07 | NULL | 0 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-23 20:13:29 | 2014-12-23 20:13:31 | 2014-12-24 03:34:06 | | | 172.16.10.10:3260,1 iqn.2010-10.org.openstack:volume-853e0bcc-05cf-484d-859d-1bcbfa46a55e 1 | CHAP Rt5PCuZJFwahV76e74u2 S6e7A2tSGcwyKvdd4PG5 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | | 2014-12-24 03:37:55 | 2015-01-09 08:58:15 | NULL | 0 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack | 1 | nova | NULL | NULL | NULL | error_deleting | detached | 2014-12-24 03:37:55 | 2014-12-24 03:37:57 | 2015-01-09 08:58:15 | | | 172.16.10.10:3260,2 iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 1 | CHAP 5mgeSdMVPxTCPMFhqc2R wv4NJM9tVLKhWpZwpaB2 | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | NULL | +---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+ 2 rows in set (0.00 sec) </pre> Attempt to delete volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e fails due to foreign key constraint: <pre> mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_glance_metadata`, CONSTRAINT `volume_glance_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`)) </pre> Check what we have in volume_glance_metadata: <pre> mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-23 20:13:31 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | container_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 3 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_ram | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 4 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 5 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | disk_format | ami | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 6 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 7 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 8 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 9 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | min_disk | 0 | | 2014-12-23 20:13:31 | NULL | NULL | 0 | 10 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL | size | 25165824 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 20 rows in set (0.00 sec) </pre> Clean volume_glance_metadata now and try to delete the volume from volumes table again: <pre> mysql> delete from volume_glance_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 10 rows affected (0.00 sec) mysql> select * from volume_glance_metadata; +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | snapshot_id | key | value | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ | 2014-12-24 03:37:57 | NULL | NULL | 0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | kernel_id | 20ee87ad-94b4-4cab-9d58-5959b7f51412 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | container_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_ram | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | ramdisk_id | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | disk_format | ami | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_name | cirros-0.3.2-x86_64-uec | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | image_id | efdf518c-c021-47f6-94b1-fe93b0fa3969 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | checksum | 4eada48c2843d2a262c814ddc92ecf2c | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | min_disk | 0 | | 2014-12-24 03:37:57 | NULL | NULL | 0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL | size | 25165824 | +---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+ 10 rows in set (0.00 sec) mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`)) </pre> Boom! We have more in volume_admin_metadata. Clean it too: <pre> mysql> select * from volume_admin_metadata; +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | created_at | updated_at | deleted_at | deleted | id | volume_id | key | value | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ | 2014-12-23 20:13:34 | NULL | NULL | 0 | 1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | readonly | False | | 2014-12-23 20:13:34 | NULL | 2014-12-24 03:34:06 | 1 | 2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | attached_mode | rw | | 2014-12-24 03:37:59 | NULL | NULL | 0 | 3 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | readonly | False | | 2014-12-24 03:38:00 | NULL | 2015-01-09 08:58:14 | 1 | 4 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | attached_mode | rw | +---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+ 4 rows in set (0.00 sec) mysql> delete from volume_admin_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 2 rows affected (0.00 sec) </pre> Now again back to cleaning volumes table and it works this time: <pre> mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e'; Query OK, 1 row affected (0.00 sec) </pre> Repeat the process for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26: <pre> mysql> delete from volume_admin_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 2 rows affected (0.00 sec) mysql> delete from volume_glance_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 10 rows affected (0.01 sec) mysql> delete from volumes where id='876771b2-04e6-4e80-81ba-6102deb6fa26'; Query OK, 1 row affected (0.00 sec) mysql> \q Bye </pre> And it's all clean now: <pre> [stack@devstack devstack]$ cinder list +----+--------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +----+--------+--------------+------+-------------+----------+-------------+ +----+--------+--------------+------+-------------+----------+-------------+ </pre>

Return to Devstack. After bringing laptop back from sleep mode instance is in 'SHUTOFF' status and volume is in 'error deleting' status..