I can see why the last idea you shared would be attractive in your situation where you’ve designed your system around the original behaviour of data sources, but our research indicates that a significant number of authors don’t consider it strange for Terraform to detect changes to data sources while planning because the primary purpose of that feature is to respond to changes outside of the configuration: it effectively says “if this other thing changes, dynamically change my configuration in response so I don’t have to”.
That’s different than disabling refresh because when using Terraform robustly it is often valid to assume that a managed resource will still be the same as it was last time, because nothing should be changing those objects except the current Terraform configuration. The refreshing Terraform does by default is just in case something weird has happened, and some teams prefer to set things up so that weird things cannot happen and then turn off refreshing to speed up planning, because they trust it will never yield anything useful.
Our research into batching (many years ago now, unfortunately) suggested that enough APIs commonly used with Terraform had some means of batching that should benefit at least situations involving reading a number of objects of the same type, and in some cases reading many objects of different types (as is the case for any GraphQL API, but also with multipart request proxy endpoints wrapping some REST APIs).
You are right that it can’t solve everything, but I don’t think it really needs to: there are certain object types whose usage patterns tend to encourage large numbers of objects, such as anything which scales with number of people in an organisation. But there are also plenty of things where a typical configuration only interacts with one or a few objects of the same type, and batching would offer only a modest improvement for those anyway.
For your existing modules today, it seems like you might benefit from some refactoring so that your shared modules accept as input the relevant results of loading the groups, rather than each one reading the same information. In other words, that’s manually implementing the sort of batching you described in the first point within your module, rather than Terraform doing that automatically.
It would be interesting to explore Terraform doing that automatically, and it might even just come as a nice side-effect of batching because Terraform Core would already need to be comparing multiple pending reads to notice when they are batchable, and noticing that two are exactly identical is in theory an easy special case to implement once we’re already comparing and bucketing all of the reads anyway.
But in today’s Terraform it remains an author’s responsibility to trade off convenience vs. performance, just as is true in many other languages: while it is often useful to encapsulate all of the queries a given component needs inside that component, developers often need to compromise to centralise the lookups of some commonly-used values and pass them in as inputs to the other components, even though that does weaken the encapsulation by exposing which components are depending on that shares data.