Bug Hunt #1: Orphaned Amazon S3 event notifications for AWS Lambda functions

I have been doing a lot of infrastructure-as-code (IaC) work for my team, recently. We are heavy users of Amazon Web Services (AWS) and use Terraform as our main IaC tool.

Over the past few weeks, I have spun up and torn down my fair share of AWS Lambdas using Terraform, and the AWS bug(s) highlighted here were spotted during this time.

The main issue is that Amazon S3 notifications created through AWS Lambda triggers on the AWS Lambda screen are not properly cleaned up when Lambda functions are deleted.

Also highlighted herein is that the use of Lambda function names (or more generally, resource names, as opposed to their unique identifiers [ARNs]) for navigation in the AWS Console is problematic from a user experience point of view.

Steps to reproduce

Executing actions primarily from the AWS Lambda screen

1. Create an Amazon S3 bucket (default configurations are sufficient)
2. Create an AWS Lambda function (Python3.8 runtime; execution role having S3 read-only permissions). You’ll need to use its name later, so keep it in mind.
3. Create a Lambda trigger (via the AWS Lambda screen) that invokes the Lambda function when “All object create events” events are emitted from the S3 bucket.
4. Ensure that the Lambda trigger has been successfully been added by reviewing the Lambda function’s triggers on the Configuration tab of the AWS Lambda screen and navigating to the Event notifications section of Properties tab for the S3 bucket to confirm the presence of an event notification for the Lambda function.
5. Return to the Configuration tab on the AWS Lambda screen and delete the Lambda trigger for S3 that was created in Step 3.
6. Review the Event notifications section of the Properties tab for the S3 bucket. Notice that the event notification is gone. This is expected.
7. Return to the Configuration tab on the AWS Lambda screen and repeat Steps 3 and 4.
8. Via the Actions drop-down menu on the AWS Lambda screen, delete the Lambda function. You should see a prompt for confirmation that reads, “Deleting a function permanently removes the function code. The related logs and roles are retained in your account.” Confirm to delete.
9. Review the Event notifications section of the Properties tab for the S3 bucket. Notice that an event notification referencing the deleted Lambda function is present. This is NOT expected. My expectation is that the event notification should have been deleted too.
10. Click the link to the Lambda function referenced in the event notification. It will try to navigate you to the Lambda function’s screen, fail, and redirect you to a list of AWS Lambda functions in your current region.
11. Repeat Step 2 using the same Lambda function name. Notice that the Lambda function doesn’t have any triggers even though there is an event notification for an S3 bucket referencing a Lambda function with the same name. Perhaps this is working as designed since AWS resources are typically identified by the ARNs and not their names.
12. Return to the Event notifications section of the S3 bucket’s Property tab and click the link to the Lambda function. It will redirect you to the AWS Lambda screen for the lambda you created in Step 10. Here we have a case were the event notification may be associated through the Lambda function’s ARN but uses the function’s name for navigation in the AWS Console. Confusing!
13. For extra measure, add a file to the S3 bucket and observe that the recreated Lambda does not trigger.

Executing actions primarily from the Amazon S3 screen

1. Remove the orphaned notification from the S3 bucket’s event notifications list.
2. Create a new event notification (configured for “All object create events”) and attach it to the recreated Lambda function from Step 11 of the previous section.
3. Review the Lambda function to confirm that a corresponding Lambda trigger is present.
4. Via the Actions drop-down menu on the AWS Lambda screen, delete the Lambda function.
5. Recreate a Lambda function with the same name and configuration.
6. Notice that an AWS Lambda trigger isn’t present for the newly created Lambda function but an event notification is present in the Event notifications section of the S3 bucket’s Properties tab for the recently deleted Lambda function. This seems reasonable since it was not the Lambda function that added the notification, but it’s still confusing that the link in the notification directs users to the new Lambda function which is clearly not the same Lambda function to which the notification was associated.
7. Review the issues highlighted in Steps 9–12 of the previous section.

I have already brought these issues to the attention of the AWS team and hope the issues get fixed real soon.

Thank you for reading!

More Computing resources

Watch videos covering a variety of topics in Computing at OnelTalksTech.com