We are creating multiple clusters on demand for testing purposes. After testing is done, the clusters are destroyed. We may create and destroy around 15-20 clusters per day.
Recently we noticed that on new EMR clusters, autoscaling policies fail to attach with reason “Failed to provision the AutoScaling policy: The maximum limit of 500 scalable targets for service namespace elasticmapreduce has been reached”.
Listing scalable targets with “aws application-autoscaling describe-scalable-targets --service-namespace elasticmapreduce” shows 500 registered scalable targets belonging to old, non-existing EMR clusters.
Is this expected behavior? Should we manually deregister scalable targets after destroying EMR resources?
- Create an EMR cluster with an autoscaling policy attached to an instance group
- Destroy created resources
- Run “aws application-autoscaling describe-scalable-targets --service-namespace elasticmapreduce”, this will show a registered scalable target from the destroyed EMR cluster