Difficulty using JSON as the language for my configuration files

Terraform
1

I am a small-scale individual user of Terraform for the configuration of a few servers I run in the AWS cloud. I dislike the proliferation of data serialization formats, and believe that JSON is fully adequate for almost all purposes. So I try to standardise for myself on using JSON files. The literature of Terraform says a number of times that HCL (“native syntax”) is “easier” and " recommended". But they also say that the terraform software can use JSON files as well as HCL files; and that for automation this may be an acceptable choice.

However, the documentation of using JSON is very limited. The one link to a full spec of the HCL use of JSON in 0.12+ is broken. And there are very few examples to be found of json config files. There is also no conversion tool provided.

I have tried a number of times to set up a fairl basic set of files to create a couple of servers in a custom vpc with one private and one public subnet. But terraform does not accept my json files. I know that under terraform 0.11, with maybe a few quirks, it did. Now it does not. Neither when I convert using one of the web or command-line tools. Nor when I make all the tweakings that I can think of.

Has anyone else had this experience? Is there any source of ideas or examples that someone can suggest? I’d be most grateful for any pointers where to look. If needed I can give an example of simple json files that are rejected by terraform init and by terraform validate. But I think that the point itself is general, and clear enough.

2

Hi @henrystrick,

If you can provide some specific examples of what you tried and what happened when you tried it then I’m happy to give some pointers.

Terraform’s JSON support is primarily intended for machine-generated rather than human-generated input, but of course you can hand-write Terraform configuration in JSON format if you want, with the caveat that (as you’ve seen) the documentation generally assumes that you will learn the native syntax first and then learn how to translate it into JSON for the presumed rare cases you need it, rather than that you will aim to use the JSON syntax exclusively.

3

Hi there,

Thank you very much. It’s very kind of you to give me a hand on this. Do you have a preference for doing it in the Discussion Forum or by email? I have a good example of a small test case with a smallish example of a file in HCL, and a JSON file that is rejected by ‘terraform validate’, and that I’d want to learn how to reformulate so that it works… I’m ok either way. On the Forum it might be of interest conceivably to someone else. Though you are right that there probably aren’t that many people who do what I do.

Henry

4

Hi @apparentlymart ,

This is a concrete example of my problem with using JSON for my Terraform configuration files.

I am using Terraform v0.12.26 + provider.aws v2.66.0. I work locally on a desktop with Linux fedora 32; put up servers in AWS which are based on CentOS, at the moment mainly for running a few simple websites. Security is quite important for me. I want to automate my server provision set-up for the longer term.

I set up a small basic “test ground” which creates one web server, using IPv4 and IPv6, with a NAT instance, custom gateways, custom VPC, and a public and private subnet.

The config I created is composed of the following files, which are I think fairly self-explanatory:
main.tf.json
vars.all.tf.json
terraform.tfvars.json
vpc_and_gateways.tf.json
subnet_public.tf
subnet_private.tf
web_server.tf.json
web_secgroup.tf
NAT_instance.tf.json
NAT_secgroup.tf

Originally I had (my full version of) this with all the files created in JSON. I could not make that work - it basically stalled at the terraform validate stage.

I have now converted the main JSON files which produced error messages to HCL; and created a mixed JSON / HCL setup as listed above. This does work, and I have deployed the server, the NAT instance, and the different customised networking components - all verifiable in the console.

I have also identified that some of my problems were a struggle with the different interpolation syntax styles: especially in terraform 0.12, in HCL you hardly use the old forms with ${ } , and also dispense with many of the quotation marks, e.g. both for aws_vpc.vp3333.id and for var.port_http or var.ami_id However, even under 0.12, in the JSON config files you need to use the syntax with ${ } , or else the files get rejected by terraform validate.

These interpolation problems I think I now have under control, and I can produce valid expressions in JSON as well as in HCL.

My current main example problem is the following - illustrated by trying to convert one file from HCL to JSON.

I tried to convert the web_secgroup.tf file to JSON - while leaving the rest of the files - which produce a functioning deployment. The HCL file was as follows:

web_secgroup.tf

resource "aws_security_group" "Web_server_SG333" {
  name           = "sec group web server SG333"
  vpc_id         = aws_vpc.vp3333.id
  revoke_rules_on_delete = true

  ingress {
    description   = "http-4"
    from_port     = var.port_http
    to_port       = var.port_http
    protocol      = "tcp"
    cidr_blocks   = ["0.0.0.0/0"]
  }

  ingress {
    description        = "http-6"
    from_port          = var.port_http
    to_port            = var.port_http
    protocol           = "tcp"
    ipv6_cidr_blocks   = ["::/0"]
  }

  egress {
    description   = "http-4"
    from_port     = var.port_http
    to_port       = var.port_http
    protocol      = "tcp"
    cidr_blocks   = ["0.0.0.0/0"]
  }

  egress {
    description        = "http-6"
    from_port          = var.port_http
    to_port            = var.port_http
    protocol           = "tcp"
    ipv6_cidr_blocks   = ["::/0"]
  }

  ingress {
    description  = "ssh-in-fr-outside-4"
    from_port    = var.port_ssh
    to_port      = var.port_ssh
    protocol     = "tcp"
    cidr_blocks  = [
      "193.36.0.0/27",
      "185.159.0.0/27"
    ]
  }

  ingress {
    description      = "ssh-in-fr-outside-6"
    from_port        = var.port_ssh
    to_port          = var.port_ssh
    protocol         = "tcp"
    ipv6_cidr_blocks = ["2a00:c98:2050:a02f:xxxx:yyyy:6b38:2ba/128"]
  }

  tags  =  {
    Name = "Security group for web server SG333"
  }
}

I have experimented with various different approaches of putting in more or fewer brackets and braces. Especially if you would use JSON files in an automated environment, I would expect there to be some conversion tool or tools. I would dearly like to have one at my disposal. I have found an old (and perhaps by now unmaintained?) Linux tool called json2hcl , which converts in both directions. However, this is clearly pre-0.12, and works badly with variable expressions… However, I can utilise it to some extent. One of the existing web tools converting between HCL and JSON was worse.

So using mainly this json2hcl tool, and adjusting the expressions, I have converted the HCL above to the following JSON file:

web_secgroup.tf.json

{
  "resource": [
    {
      "aws_security_group": [
        {
          "Web_server_SG333": [
            {
              "egress": [
                {
                  "cidr_blocks": [
                    "0.0.0.0/0"
                  ],
                  "description": "http-4",
                  "from_port": "${var.port_http}",
                  "protocol": "tcp",
                  "to_port": "${var.port_http}"
                },
                {
                  "description": "http-6",
                  "from_port": "${var.port_http}",
                  "ipv6_cidr_blocks": [
                    "::/0"
                  ],
                  "protocol": "tcp",
                  "to_port": "${var.port_http}"
                }
              ],
              "ingress": [
                {
                  "cidr_blocks": [
                    "0.0.0.0/0"
                  ],
                  "description": "http-4",
                  "from_port": "${var.port_http}",
                  "protocol": "tcp",
                  "to_port": "${var.port_http}"
                },
                {
                  "description": "http-6",
                  "from_port": "${var.port_http}",
                  "ipv6_cidr_blocks": [
                    "::/0"
                  ],
                  "protocol": "tcp",
                  "to_port": "${var.port_http}"
                },
                {
                  "cidr_blocks": [
                    "193.36.0.0/27",
                    "185.159.0.0/27"
                  ],
                  "description": "ssh-in-fr-outside-4",
                  "from_port": "${var.port_ssh}",
                  "protocol": "tcp",
                  "to_port": "${var.port_ssh}"
                },
                {
                  "description": "ssh-in-fr-outside-6",
                  "from_port": "${var.port_ssh}",
                  "ipv6_cidr_blocks": [
                    "2a00:c98:2050:a02f:xxxx:yyyy:6b38:2ba/128"
                  ],
                  "protocol": "tcp",
                  "to_port": "${var.port_ssh}"
                }
              ],
              "name": "sec group web server SG333",
              "revoke_rules_on_delete": true,
              "tags":
                {
                  "Name": "Security group for web server SG333"
              },
              "vpc_id": "${aws_vpc.vp3333.id}"
            }
          ]
        }
      ]
    }
  ]
}

How can I make a JSON file that does the same as the HCL file, and that is accepted by terraform? If you can answer that question, I can probably generalise from there to my other “problem files”, and revive my project and my favoured way of working.

I know that this conversion is a bit top-heavy in the alternation between and {} brackets. And can simplify that myself. I have tried a number of variations, but I did not get any of them to work.

The error message produced by terraform validate to the full set of files, including above web_secgroup.tf.json file, is as follows:

Error: Incorrect attribute value type

  on web_sec_group.tf.json line 8, in resource[0].aws_security_group[0].Web_server_SG333[0]:
  
  << I omit the print-out of the egress portion of the file - lines 8-27 >>
  
  Inappropriate value for attribute "egress": element 0: attributes "ipv6_cidr_blocks", "prefix_list_ids", "security_groups", and "self" are required.


Error: Incorrect attribute value type

  on web_sec_group.tf.json line 28, in resource[0].aws_security_group[0].Web_server_SG333[0]:

  << I omit the print-out of the ingress part of the file; lines 28-66 >>

  Inappropriate value for attribute "ingress": element 0: attributes "ipv6_cidr_blocks", "prefix_list_ids", "security_groups", and "self" are required.

I hope the above reasonably describes the problem I encounter. It may be that the solution is simple. But it left me stumped. I have spent probably hours on the clearly important documentation about JSON configuration syntax, including the bit on Nested block mapping, which I believe probably contains the answer that I am looking for; but I have not been able to work it out with that as help. There seems to be a reference document of the full HCL2 JSON syntax somewhere in Github, but the link to that is broken. Otherwise I would have tried to dive into that, too.

I am most grateful for your or anyone’s help with this.

5

Hi @henrystrick,

It looks like you’ve got most of it figured out here. For those egress and ingress blocks, unfortunately I think you’ve run into the “Attributes as Blocks” situation, which was an awkward compromise made in Terraform 0.12 to work around some providers that were previously using Terraform language features in an unsupported way, relying on some missing validation rules in Terraform 0.11 and earlier.

Specifically, note the In JSON Syntax section at the end of the page. Because ingress and egress are both being processed in this unusual processing mode, the Terraform language engine is expecting you to use JSON expression mapping for these, instead of the block mapping as normal, to allow for the case where it would be set explicitly to [] to say “no rules”.

The upshot of this is that you need to write out these particular blocks as an array of objects with all of the arguments specified, because attribute values in Terraform do not support “optional” attributes. (This only works in the native syntax because the native syntax parser “cheats” and uses the explicit difference between block vs. attribute syntax to dynamically select which interpretation the author intended.)

{
  "egress": [
    "cidr_blocks": ["0.0.0.0/0"],
    "description": "http-4",
    "from_port": "${var.port_http}",
    "protocol": "tcp",
    "to_port": "${var.port_http}",
    "ipv6_cidr_blocks": null,
    "prefix_list_ids": null,
    "security_groups": null,
    "self": null,
  ]
}

Specifying the unused attributes as null like in the above should cause Terraform (or rather, the AWS provider) to accept the object as valid. The above is equivalent to the following native syntax:

  egress = [
    {
      cidr_blocks      = ["0.0.0.0/0"]
      description      = "http-4"
      from_port        = var.port_http
      protocol         = "tcp"
      to_port          = var.port_http
      ipv6_cidr_blocks = null
      prefix_list_ids  = null
      security_groups  = null
      self             = null
    }
  ]

Notice that this is using egress = [{ ... }] rather than egress {}, because the provider has actually declared egress to be a single argument that takes a list of objects as its value. The block syntax is an accommodation Terraform is making to allow the more idiomatic form in the native syntax in spite of the provider’s unusual schema for this argument.

This oddity is expected to phase out in future major versions of the providers that are relying on it, once they change their configuration schemas to no longer depend on this old Terraform 0.11 bug. With that said, I don’t see an open issue in the AWS provider repository about this specific case, so it could be worth noting what you ran into there as a prompt for the AWS provider team to consider moving away from this legacy processing mode in a future major release, and allow ingress and egress to behave like normal blocks in both syntaxes.

6

Hi @apparentlymart ,

That was beautiful, your reply. Utterly satisfactory. I didn’t see that coming, although from a literal reading of the error messages, somehow it was “in there”: they now make sense too.

I have made the adjustments you mentioned, and it works. So everything was right, other than the fact that with JSON config files the terraform engine expects for “attributes as blocks” that every single attribute is given an explicit value. There remains that I think that in the documentation Terraform could really give an indication of this, specifically in the pages devoted to “JSON Configuration Syntax” and to “Attributes as Blocks” - which after all will only be read by people who are using JSON.

They write that “Everything that can be expressed in native syntax can also be expressed in JSON syntax, but some constructs are more complex to represent in JSON due to limitations of the JSON grammar”, which is a pretty strong statement. Therefore I really think it’s not asking too much that they would add some kind of general warning about this. Indeed your reply shows that it can be done; but documentation is partly expected to be helpful, no? In any case I assume that this is not your responsibility. If / when I have time, and see an opportunity, I’ll keep making this point about the documentation of JSON config files.

I am glad to be back on track with my project to document and automate my way of working. Given that these are pretty permanent fixtures of my set-up, it does not matter that one time I need to write somewhat more extensive config files. Once they are there, they will last.

And I somewhat wonder, following your explanation, about the following: If terraform can let the native syntax cheat on this point, why would it be so difficult or unreasonable that they would also let their JSON parser cheat in the same way?

I will see if I send some summary of this to the AWS provider repo team. I first want to go back to my full set of configs, and see if I can now really make everything work in JSON, or that I still encounter some other problems.

Adapting this to the route tables wasn’t completely trivial; but I did manage.

Thank you so much for your attention to this, and the quick and effective follow-up!