This bash script will delete failed volume attachments / volumes from a stack in Openstack.

Why?

Sometimes Heat can fail to delete a stack for a variety of different reasons: The volume is in use, the VM failed to stop, the volume went into an error state, or Heat sent a message to RabbitMQ which timed out, was missed or expired.

First Steps:

Determine what went wrong:

$ heat event-list $badstack

• Look at the output, usually will indicate that there was a volumeattachment failure (cannot detach volume while in a detached state)

What happens here, is that cinder will lose sync with the openstack controller and not know whether it’s volume still exists and when it tries to execute it’s api call, the volume detach comes back with an error, as volume attachment at this stage is only in the database.

• run the following commands as root (while also logged into OS)

/destroyvolumes.sh <stackname>
e.g.
/destroyvolumes.sh es-jdoe-01

The script will emit a series of GUIDs and attempts to reset their state and delete them through the cinder api. This process usually works, unless something is in a really inconsistent state, in which case you’ll have to go through the dmsetup route to delete the objects out of lvm. Look at fixingcinder guide to go further.

#!/bin/bash
volumes=''
delvol ()
{
	while (( "$#")); do
    cinder reset-state --state available --attach-status detached $1
	  cinder force-delete $1
	  shift
	done
}
getvols()
{
   openstack volume list|grep $1| cut -f 2 -d '|'|sed 's/^ //ig'
}
volumes=$(getvols $1)
echo $volumes
echo "----"
echo "$volumes"
delvol "$volumes"

BE REALLY CAREFUL with this script as it WILL DESTROY the data on that stack. Please exhaust all other avenues of repairs before attempting this fix