Ran into an issue on a terraform module I was working with that creates a security group – the input data structure isn’t flexible enough due to type constraints and I’m going to do some surgery on that. The module is not currently using any test framework (other than the fact that it’s used, and tested through use), so I thought I’d try writing one using “terraform test”.
The simplest test worked fine, but I very quickly ran into limitations in terms of what the module itself exposes and what I can get out of Terraform OOTB data sources. For instance, if I want to validate the ingress rules of the security group were created, those rules aren’t exported as outputs of the module nor do I see a data source in the terraform aws provider that exposes the rules of the data source (if I’m missing something, feel free to let me know).
Since the experimental testing support suggested we share our feedback, this is something that seems important to me – that there may be limitations on what we can easily test this way. Of course, I could certainly write my own provider, or look at other providers to see if I can make an AWS API call … but if simple things are hard to test simply, I would probably at least consider looking at some of the other frameworks before digging in too deeply here. I like the idea of having test support built into Terraform, but I can imagine this is going to be one of the areas that poses some challenges?
Anyway – just wanted to share my first experience with it as feedback, see if this is something other people have run into.
I see that the aws_security_group data source dates back to a 2018 addition:
Perhaps the AWS provider team would be willing to either add a data source corresponding to this other API action or to add additional capabilities to the existing data source to also fetch the security group rules for the action. The use-case of testing in this way might not be sufficient motivation to support it for as long as terraform test remains experimental, but I think it couldn’t hurt to open an issue to discuss it.
I will take this general feedback to the Terraform Core team though, since I expect this isn’t the only situation where writing tests using data sources would encourage adding data sources that wouldn’t necessarily be useful otherwise, and that’s definitely a design consideration to ponder: is it reasonable to impose new feature requests like this on various provider developers only to support testing, or would it be better to address it in some other way than using data sources?
Yeah, I think that’s part of what I’m pondering here. Is having a built-in test mechanism valuable enough to want to potentially add data sources whose primary use case might be testing?
Or, on the other hand, if adding all the data sources that might be required to effectively test Terraform modules sounds like a burden on maintenance and documentation and so forth, is that an argument to be made for using a test framework that’s integrated directly with something like the AWS API instead of using Terraform data sources?
Anyway, was interesting to try using this experimental test framework, but this experience has raised more questions for me. (Which is fine).