So it turns out that this was all my fault 
We use AWS IAM & EC2 authentication heavily and I wasn’t revoking the generated authentication tokens. These were (I guess by default) valid for a month so, under heavy usage, the number of tokens lying around waiting to expire was just growing.
I’m now going through all of our scripts to amend them to explicitly revoke any tokens once I’m done with them.