A variation of around 3% when making so many network calls to remote APIs isn’t really conclusive of anything. Again, without specifics I can’t say for sure, it’s possible the structure of the configuration doesn’t allow for much concurrency. Many services have rate limits for protection too that are easy to hit with large configurations, so you may only be able access so many resources within a certain amount of time.
If significant time is spent reading the remote resources, then -refresh=false could speed things up by skipping the first read. That may run into other problems however if resources aren’t correctly updated before planning (and in old versions of terraform, data sources may behave unexpectedly as well).