How to manage sensitive diffs?

My field is a TypeMap of TypeString where the key is not sensitive, but the value is sensitive. I can specify Sensitive = true in the Elem schema, but how do I manage the diffs? The underlying API returns ***** for the values which is different than the real values in the config so every apply results in a diff detected. I would like for a diff to be detected ONLY when the user changes it in the config, not because of ***** being read back during Read.