Not sure if this is a best practice but my approach to this so far has been to:
Create a terraform IAM user. It has no privileges except to AssumeRole for roles specifically provisioned for configurations (workspaces).
Depending on your needs you could use the same Roles for multiple workspaces, but we’re trying to keep them minimally privileged.
Each workspace will assume_role for the role that has only the privileges needed for that configuration.
This minimizes what can be done and means the terraform IAM account itself has no privileges—not even to ListRoles. For these keys to be useful to anyone if lost, you’d also need to know the ARN of at least one role that it can assume.
Depending on your needs you could further divide into multiple TF IAM accounts if you need to isolate more.
We are using a separate terraform configuration to manage these roles themselves using terraform (you’ll need to manually bootstrap it though…chicken vs egg).
The last remaining piece is to automatically rotate access keys in TFC… this might be done using an on-prem process (e.g. Jenkins job) to run periodically to update all the workspaces with new keys on a regular basis through the TFC API or TFE provider. Haven’t really found a better solution for that.
Would love to see the possibility of TFC supporting cross account roles natively for AWS to avoid the access key situation.
Note: this is for use with Remote runs.