Monitoring Failed Amazon RDS Snapshots: An Unexpected AWS Limitation
One of the recent tasks I worked on sounded quite simple at first.The goal was to send a Slack notification whenever an Amazon RDS or Aurora snapshot creation failed. Since AWS RDS already publishes many events through EventBridge, I assumed there would be a dedicated event for snapshot failures as well.After reviewing the AWS documentation, we found several failure events that looked promising:
RDS-EVENT-0159RDS-EVENT-0162RDS-EVENT-0484RDS-EVENT-0489
At first glance, they seemed to be exactly what we needed. However, after reading the documentation more carefully, realized that all of these events are related to snapshot export tasks to Amazon S3, not to the actual snapshot creation process.That immediately raised a question:What happens if the snapshot creation itself fails? To answer that, I tried reproducing the scenario in one of our development environments. I started creating a manual snapshot, hoping I could stop it while it was still running and force a failure. Unfortunately, AWS doesn't allow canceling an in-progress snapshot, so that approach wasn't possible.At that point, the documentation still didn't provide a clear answer, so we decided to open a support case with AWS. The response was quite surprising. AWS confirmed that Amazon RDS and Aurora do not generate an EventBridge event when a manual or automated snapshot creation fails. In other words, there is no native "snapshot failed" event that can be used to trigger an alert. AWS also suggested several possible alternatives.One option was periodically calling the DescribeDBSnapshots or DescribeDBClusterSnapshots APIs and checking whether any snapshots have a failedstatus. Another option was correlating the "creating" and "created" events and raising an alert if the completion event never arrives within an expected time window. Both approaches would work, but they also introduce additional logic or extra components. After discussing the available options, we decided to keep the solution as simple as possible. Instead of trying to detect a failure event that doesn't exist, we changed the monitoring logic to verify that at least one snapshot is successfully created every 24 hours. The alert checks a rolling 24-hour window for successful snapshot creation events. If no "created" event is found during that period, the rule waits for a three-hour grace period before triggering an alert. This avoids false alarms in cases where a snapshot is simply delayed instead of actually failing. I actually like this approach more than the original idea. In the end, the real requirement isn't to know whether a snapshot failed - it's to know that we still have recent backups. Monitoring successful snapshot creation every 24 hours answers that question directly. One interesting observation from this investigation is that AWS Backup provides native FAILED backup job events, while native RDS snapshot creation doesn't provide an equivalent failure event. I'm not saying this is intentional, but if backup failure notifications are an important requirement, AWS Backup certainly offers a more complete monitoring experience out of the box. This was also a good reminder that documentation doesn't always answer every question. Sometimes the fastest way to understand how a cloud service really behaves is to test it yourself and, when needed, ask the people who built it. Hopefully, this saves someone else the same investigation.