Deleting a volume that cinder won’t delete (Error Deleting)
if a volume goes into a state of error_deleting, you can still manually delete it, this is caused usually by amqp timeouts or a volume persisting across the reboot of a controller in a disconnected state.
- get the UUID of the volume from cinder:
[root@oscontroller openstack_scripts]# cinder list|grep web | 21663b5c-8a87-4809-ba45-591a5291fded | error_deleting | - | web-loadbalancer-03-volume_04-rbfttdc7bysb | 1 | - | false | False | | [root@oscontroller openstack_scripts]#
- show the volume ID of the UUID
[root@oscontroller openstack_scripts]# lvs|grep 21663b5c-8a87-4809-ba45-591a5291fded
volume-21663b5c-8a87-4809-ba45-591a5291fded cinder-volumes -wi-ao---- 1.00g- locate the volume in dmsetup
[root@oscontroller openstack_scripts]# dmsetup ls
...
cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded (253:48)
...- Attempts to remove the volume will fail
[root@oscontroller openstack_scripts]# dmsetup remove cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded
device-mapper: remove ioctl on cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded failed: Device or resource busy
Command failed
[root@oscontroller openstack_scripts]# dmsetup remove cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded --force
device-mapper: remove ioctl on cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded failed: Device or resource busy
Command failed
[root@oscontroller openstack_scripts]#- If the standard lsof method doesn’t return anything, then you’re going to have to dig a little deeper
[root@oscontroller openstack_scripts]# lsof |grep 253,48
[root@oscontroller openstack_scripts]#- Let’s use dmsetup to get detailed info about the device:
[root@oscontroller openstack_scripts]# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
mpathe 253 5 L--w 0 1 0 mpath-360001ff07dd740000000001089640010
cinder--volumes-volume--d65a6fe2--8889--416c--926e--51334a0779e0 253 7 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rUFMSkvqYHrYSlshwdhV8MBujWu3PNmPk
mpathd 253 3 L--w 0 1 0 mpath-360001ff07dd7400000000003893f0003
cinder--volumes-volume--73882cb1--7023--495b--b4a4--fa5f473d6b9a 253 12 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rQRsX9gUBdpprhsKB2mdznHzbMANCkX40
cinder--volumes-volume--7060d459--bcc0--491c--882f--c49e7f12e58e 253 21 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8r1GGnb7rYJ2VEuuhUsDGKBoIBN2vsVDvl
cinder--volumes-volume--1dc5515c--4e62--4a45--a656--0cdf17c94f2e 253 17 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rkeR97GesoZLLneRSerHz08bRWiAOyoic
mpathc 253 2 L--w 1 1 0 mpath-360001ff07dd740000000000189390001
cinder--volumes-volume--4d6e5bfe--5cc9--4b9b--b5d9--66b782b48831 253 15 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rGRDaUtMwmdv43a5SfCOc4m8WLBuCRm2H
vg_mdt0000_testfs-mdt0000 253 82 L--w 0 1 0 LVM-PJFpQ2UKlbdu7vQ6EMw016LCsOzluiDdTcW2N2iN22TpexLMf10HcFBAYOyZQ92M
cinder--volumes-volume--41c70851--cc37--42cf--befc--a6867007090b 253 8 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8r2TXB8QSAc7hcSJc1v27ONyMkxUIMl7Yu
mpathb 253 4 L--w 1 1 0 mpath-360001ff07dd740000000000489400004
cinder--volumes-volume--05d93483--af21--4c13--9b3a--ec13e583492c 253 20 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8riYpPCLoe81h2rOK9365MZcJO6ri8vXkc
cinder--volumes-volume--6a4028e8--936e--4555--870e--93fb1b7e7678 253 9 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rvnDrNXJOdtXlveeX3nrjaV6yLMkRfzgc
mpatha 253 6 L--w 22 1 0 mpath-360001ff07dd740000000000089380000
cinder--volumes-volume--9f38f0c2--bed0--4be9--bf4c--bfb6792f048c 253 70 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rZTpsBQIjuGb433jipvujyVQtKZXxuofR
rhel-swap 253 1 L--w 2 1 0 LVM-429KB8FtlWhsXCFJEtCfntcOxSgoYjCTTfm5Ehnbwi1MFsxdPfsFUov6Id3r0HMb
cinder--volumes-volume--4d24554c--2e55--4b7b--954e--e9ff6d721526 253 11 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rgVX7r1uJo50JvrhCGIAEQJkFfmFRbS6e
rhel-root 253 0 L--w 1 1 0 LVM-429KB8FtlWhsXCFJEtCfntcOxSgoYjCTig62buraEAjw3IKO63MPcdicIJyKpMqK
cinder--volumes-volume--689349b3--3e9e--4552--bf7d--8a3b9b782524 253 19 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rtsyNBOfMBatcbHdUzTqVcGYtcKsUYupO
cinder--volumes-volume--77a0dc92--67c3--40c8--bb1c--4bb9a104b1d3 253 18 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rJGUwKO7AI7FN2HYLHreKVlWvRVUe9rAW
cinder--volumes-volume--4e6ff8b2--6e5d--4c55--acd7--3fa0412724d3 253 10 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rhfBA1GFqgXsk9LE3ZDeUFPN3qCe8DpGS
cinder--volumes-volume--7723481d--4f5f--4aff--9a0e--a8a35d45f27d 253 71 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rIcPPFhznJLf5PWOmNUB2gIYt9tluoepx
cinder--volumes-volume--6d70b2b9--1a14--48c6--ae38--268a5a9821e3 253 13 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rNP22qLRrXmRgQjbFpgp3PVKqcY1jXWeE
cinder--volumes-volume--e04038f7--e56a--4a03--b605--31cbd1e0bc0a 253 14 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rDkJ5GSbaWyi41cl4UFUYSBkIhDvHRGLE
cinder--volumes-volume--fb340945--238f--4efb--b557--6811c16fd6ad 253 77 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rkNaZLWUg0CxY0ZmDxOkVRAum2przPZG4
cinder--volumes-volume--fdf1ae6a--6d8a--46ea--9a4c--aa37be989b95 253 76 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rg1zr1dNkeTAIy1eURNoQuz2EXrRF5nSh
cinder--volumes-volume--af875a34--54e9--4b10--874b--16b12f1a0ad5 253 69 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8ruKzEKLYpNEZnC3KTNHgGdkrA9cr19WTp
cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded 253 48 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rHrQD5ZVcAq1rlTUaFAne8pwSUoUH2MVb
cinder--volumes-volume--19f7f56c--f6e1--492a--8eb1--7099ec6c217a 253 73 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rrqU4HeAHgGd29pLrkTinI5C0T1hQy10w
data-logstash 253 78 L--w 0 1 0 LVM-CcodCYUBq56oekj0IuUTV0PNjk8BmR1Pio5QFkv8keVipA7UBruBg6051P9W1bt7
cinder--volumes-volume--a35355e7--38c6--475d--919f--ade86b870398 253 74 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rufMM6cKZh9dDsQpYfi981CucSRuPuVYE
cinder--volumes-volume--bb095dc9--a0d0--4f65--af28--0e41c917d430 253 16 L--w 0 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8raOyl0t27i4xzKRDK5P0rDaZvzJajC6QG
cinder--volumes-volume--0acf8867--c2da--4c6a--aa2c--b1d1cab6d37d 253 75 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rWnbQZHl51v13fAxgnFEvIBTcTSLu48Q5
cinder--volumes-volume--aa815ec3--caf0--4626--b38d--ef0ea0f15eca 253 72 L--w 1 1 0 LVM-idiuemQdSdchu7xlK6aEpLS3CewFde8rP1JKg1FC79NFkZVBstdDtBitZvvjPFdQ
vg_mgs-mgs 253 79 L--w 0 1 0 LVM-sCiBODrC2M84TMd70mW5YfD7aoaMFrI7nx6r1Briqt66cOBfX8hVdKEd3EarCPGs
[root@oscontroller openstack_scripts]#Notice here that the Device Major,Minor are 253,48….
- lets look at whats holding the deive ‘open’
[root@oscontroller openstack_scripts]# ls -la /sys/dev/block/253\:48/holders/
total 0
drwxr-xr-x. 2 root root 0 Jan 24 11:57 .
drwxr-xr-x. 8 root root 0 Jan 24 11:55 ..
lrwxrwxrwx. 1 root root 0 Jan 29 10:59 dm-82 -> ../../dm-82There you have it, device dm82…
- Let’s remove that ….
[root@oscontroller openstack_scripts]# dmsetup remove /dev/dm-82
[root@oscontroller openstack_scripts]#- So far so good, no lets remove the cinder lv device
[root@oscontroller openstack_scripts]# lvremove /dev/mapper/cinder--volumes-volume--21663b5c--8a87--4809--ba45--591a5291fded
Logical volume "volume-21663b5c-8a87-4809-ba45-591a5291fded" successfully removed
[root@oscontroller openstack_scripts]#- Now you should be able to delete the volume attachment in cinder using the UUID:
[root@oscontroller openstack_scripts]# cinder reset-state --state available --attach-status detached 21663b5c-8a87-4809-ba45-591a5291fded
[root@oscontroller openstack_scripts]# cinder force-delete 21663b5c-8a87-4809-ba45-591a5291fded
[root@oscontroller openstack_scripts]# cinder list
+--------------------------------------+-----------+------------------+-------------------------------------------+------+-------------+----------+-------------+--------------------------------------+
| ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------------------+-------------------------------------------+------+-------------+----------+-------------+--------------------------------------+
| 05d93483-af21-4c13-9b3a-ec13e583492c | available | - | web-apache-01-volume_10-pxxawvupq3fl | 2 | - | false | False | |
| 0acf8867-c2da-4c6a-aa2c-b1d1cab6d37d | in-use | - | web-database-01-volume_01-r5ofctuxd2v7 | 1 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| 19f7f56c-f6e1-492a-8eb1-7099ec6c217a | in-use | - | web-database-01-volume_04-4j52qdsmo46p | 1 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| 1dc5515c-4e62-4a45-a656-0cdf17c94f2e | available | - | web-apache-01-volume_08-cx7cd6vbmnsj | 2 | - | false | False | |
| 4d6e5bfe-5cc9-4b9b-b5d9-66b782b48831 | in-use | - | web-apache-01-volume_05-yazcqgg3znqt | 2 | - | false | False | 0600b3ba-f648-4046-b7ff-c3033586ecc0 |
| 689349b3-3e9e-4552-bf7d-8a3b9b782524 | error | - | web-apache-01-volume_02-yaytxcsazkm3 | 2 | - | false | False | 0600b3ba-f648-4046-b7ff-c3033586ecc0 |
| 6d70b2b9-1a14-48c6-ae38-268a5a9821e3 | available | - | web-apache-01-volume_09-n7qqlq75i4j7 | 2 | - | false | False | |
| 7060d459-bcc0-491c-882f-c49e7f12e58e | error | - | web-apache-01-volume_03-npodzgzztwlg | 2 | - | false | False | 0600b3ba-f648-4046-b7ff-c3033586ecc0 |
| 73882cb1-7023-495b-b4a4-fa5f473d6b9a | available | - | web-apache-01-volume_06-gesjmockzavg | 2 | - | false | False | |
| 7723481d-4f5f-4aff-9a0e-a8a35d45f27d | in-use | - | web-database-01-volume_08-xapbsvviejo3 | 2 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| 77a0dc92-67c3-40c8-bb1c-4bb9a104b1d3 | error | - | web-apache-01-volume_04-yfjq4pxp7ope | 2 | - | false | False | 0600b3ba-f648-4046-b7ff-c3033586ecc0 |
| 9f38f0c2-bed0-4be9-bf4c-bfb6792f048c | in-use | - | web-database-01-volume_02-34fnndo6eqdy | 1 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| a35355e7-38c6-475d-919f-ade86b870398 | in-use | - | web-database-01-volume_09-jwyre6g7ejpo | 2 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| aa815ec3-caf0-4626-b38d-ef0ea0f15eca | in-use | - | web-database-01-volume_05-up3mq3xgxvuz | 1 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| af875a34-54e9-4b10-874b-16b12f1a0ad5 | in-use | - | web-database-01-volume_06-p2xgombnczaw | 2 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| bb095dc9-a0d0-4f65-af28-0e41c917d430 | error | - | web-apache-01-volume_01-6mw5yeay2f5q | 2 | - | false | False | 0600b3ba-f648-4046-b7ff-c3033586ecc0 |
| e04038f7-e56a-4a03-b605-31cbd1e0bc0a | available | - | web-apache-01-volume_07-k2rrfc3n4hnq | 2 | - | false | False | |
| fb340945-238f-4efb-b557-6811c16fd6ad | in-use | - | web-database-01-volume_07-wbcocgcimis3 | 2 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
| fdf1ae6a-6d8a-46ea-9a4c-aa37be989b95 | in-use | - | web-database-01-volume_03-iesuzvejfdxd | 1 | - | false | False | 2e6318f6-3057-4098-a28e-a9a670448a58 |
+--------------------------------------+-----------+------------------+-------------------------------------------+------+-------------+----------+-------------+--------------------------------------+
[root@oscontroller openstack_scripts]#- Note that the volume no longer appears in the cinder list; the stack should delete properly Now
Discover more from Christine Alifrangis
Subscribe to get the latest posts sent to your email.