Nested resources

Terraform Providers Plugin Development
1

Hi! There are … a lot of open github issues related to this. A lot of discussion of what not to do, but very little of what TO do, or how stuff will behave in different circumstances. Was thinking I could lay out the problem I am looking at and see if anyone can flesh some stuff out.

Am looking at the digital ocean kubernetes cluster resource. The DO api reports “node pools” as a part of the kubernetes cluster in a nested data structure. You actually can not create a cluster without having at least one node pool in the initial request. Once they are created, you can create/delete/modify the pools separately with the node pool api, but during creation that is rolled up in the cluster.

The current design of that module requires you to create exactly one node pool as a nested block on the cluster, and then you can create as many separate pools as you want outside of that. This works with the api, but the problem is now that original node pool has some magic properties. For example, something that requires a replacement of the pool winds up replacing the whole cluster (while this is not how the api works). They actually apply a label in the api so that the resource can match the node pool in its config to the one it created.

Now, not knowing anything about terraform schemas, it’d be nice to define node pools and a cluster. On create, it would merge them, and on updates/changes it would handle the pools separately. But I cannot figure out a good way to do that with the Schema. If NodePool is a separate resource, being able to merge them together during the ‘Create’ operation would work. I don’t think that’s a thing though?

Alternatively I considered using a TypeSet on the Cluster. However, it looks like there is no way with TypeSet to update individual items in the set vs adding/deleting items, and there are a bunch of attributes that can be updated without replacing.

I could use TypeList - but the API doesn’t say anything about ordering. I feel like I could sort manually to make it work, but I really don’t understand how the underlying system handles things like ordering and missing items int he Read methods.

So- are there any more detailed resources for the best way to approach stuff like this? :slight_smile:

2

I can confirm that’s not really an option. Terraform works on a resource-based model, so we can’t bundle up discrete resources in one ur-resource, unfortunately.

All of your conclusions above match my understanding. The best advice we have at the moment is a healthy “it depends”, which I’m very aware is unsatisfying. Sorry :frowning: We’re working on having more general purpose guidance for this, but it requires some of the legacy and compatibility layers that the SDK is carrying around at the moment.

If you haven’t, I’d suggest looking at the google_container_cluster and google_container_node_pool resources, which look (on the surface, at least) to have the same data model. Ditto for aws_eks_cluster and aws_eks_node_group. And if I’m not mistaken, azurerm_arm_kubernetes_cluster and azurerm_arm_kubernetes_cluster_node_pool.

Hopefully you can glean some insights from some of the precedents they’ve set?