CloudFormation provides a language for provisioning resources in AWS. In a text file you can model and provision your infrastructure. And the service makes it even easier to clean up resources.
We can specify our entire infrastructure in CloudFormation as a unit called a “stack.” And you create it with a single create command that includes the specification for the infrastructure in a text file called a “template”. Then instead of hunting down orphaned EC2 instances or the odd S3 object, you can destroy the entire stack in a single command.
The CloudFormation service even calculates the creation and deletion order for you. It sounds great. And it is. But there’s a common case that CloudFormation doesn’t handle.
The Problem
The CloudFormation service won’t delete an S3 bucket that contains objects. This means that if your stack wrote to a bucket and you didn’t manually delete the object before deleting the stack then it will fail. You will see the stack status marked at “DELETE_FAILED”.
Requiring we manually delete objects from buckets can be onerous for spinning up the infrastructure for tests or demos. We can still handle the non-empty bucket with the magic of CloudFormation.
The Solution
My solution scripts the deletion of all objects in the bucket prior to attempting to delete it. We include the logic in the CloudFormation template.
It creates a Lambda with Python code which first deletes all object versions. Then it deletes the objects themselves. We don’t delete the bucket itself with our Lambda, because we will let the delete operation of the CloudFormation stack handle that.
Overview of the Custom Resource
In our case rather than using the S3 bucket resource which CloudFormation will call to delete, we’ll make a custom resource. The custom resource will use a Lambda to delete the bucket’s contents.
If we let CloudFormation attempt the delete event directly on S3 buckets then it would fail when the buckets contained objects. Eventually it would timeout and ultimately the CloudFormation stack delete operation would fail.
Instead, our custom resource will handle events through a Lambda. The delete event will clean-up the bucket of all objects and versions before deleting. Then it will send the event response from the Lambda.
You can see the life-cycle of our custom resource in the diagram below.
Declare the Resource
We start by declaring our Bucket resource. And we reference that with a custom resource.
We’re writing our Lambda inline rather than pulling code stored in an S3 bucket. You can read more about the advantages of writing inline when interacting with CloudFormation custom resources here.
Find Object Versions
Our function code first confirms that the bucket is found. Then it checks whether versioning is enabled. If it finds that versioning is enabled then it gets a list of object versions in a paginator using the boto S3 client.
Delete Object Versions
We now iterate the pages of object versions and delete any delete markers found in the page first. Then we iterate object versions and delete them in succession.
Delete Bucket Contents
With the delete markers and versions gone, we delete the contents of the bucket with this loop. And finally report that the bucket is empty.
Send the Response
We send CloudFormation a success response, so it will stop waiting. It will continue to wait on the bucket deletion unless we send the response.
Let’s look at the complete event handler to see how that works.
You can see that we try to delete the bucket with our custom function first. If there’s no exception then we send the success response. And if there’s an exception then we print it and send the “FAILED” response.
Conclusion
You can use my project as a starting point whenever you want the CloudFormation stack to delete to include S3 buckets regardless of their contents. I’ve found this need come up often in my development. And now this approach will let me easily remove all traces of an infrastructure in my AWS account.
I hope this helps you learn, adapt, and improve.
You may also find the references below useful as I did when creating this project.
References
My github repo for this project:
https://github.com/drumadrian/custom-cloudformation-bucket-cleanup
Custom Resources in CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
AWS Lambda Function Code in CloudFormation:
Custom Resource Request Types in CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-requesttypes.html
Deleting Object Versions in S3:
https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingObjectVersions.html
Credits
The following solutions were helpful in crafting something of my own
Adrian G. Drummond 