Azure scale-set drifts

I provision linux VMs as self-hosted Azure Pipeline Agents in a scale-set in Azure via terraform.The scale-set gets changed dynamically by azure, which leads to drifts in terraform’s state. Changes like the number of instances in the scale-set or some tags which get added to the scale-set. I managed to get some of the changes ignored by terraform using the lifecycle block, but running terraform plan time to time shows new changes done outside terraform. I could not find a documentation about the dynamic changes Microsoft applies to a scale-set, so I could ignore them accordingly. Currently I use ignore_changes for number of the instances and two specific tags.
Has anyone an idea, what would be the best way to overcome this problem?