Use of "aws_security_group_rule" vs inline definition. What do you prefer?

I used to prefer the separate resource like aws_iam_role_policy_attachment , aws_security_group_rule. This allows to extend the previously defined security group or IAM policy.

But recently I realise that this allow people to add security rule or IAM policy by hand which wont be captured by terraform. Since its invisible to Terraform, it wont be removed by terraform in next plan.

This make it difficult to capture the drift in the infrastructure and possibly the security holes created by hand.
This also makes it difficult to know whether the infrastructure is working due to the automation or due to existence of some manual rules (i.e. automation gaps).

Would like to know what approach community is taking on this?