Hclwrite attribute object with reference values

Terraform
1

I wrote a reply to this old post:
Hclwrite(v2) a object with a reference value - Terraform - HashiCorp Discuss

In regards of the issue presented there. However, instead of reviving that old thread I thought it perhaps would be wise to create a new one. With that context and babbling out of the way, here goes the question:

Using the hclwrite package, I need to create the following custom_data object attribute:

post-processor "manifest" {
  custom_data = {
    subscription        = "${var.subscription_id}"
    resource_group      = "${var.resource_group}"
    gallery_name        = "${var.gallery_name}"
    image_name          = "${var.image_name}"
    image_version       = "${var.versionToBuild}"
    source_version      = "${var.info_source_version}"
    source_branch       = "${var.info_source_branch}"
    source_repo         = "${var.info_source_repo}"
    gallery_artifact_id = "/subscriptions/${var.subscription_id}/resourceGroups/${var.resource_group}/providers/Microsoft.Compute/galleries/${var.gallery_name}/images/${var.image_name}/versions/${var.versionToBuild}"
    managed_artifact_id = "/subscriptions/${var.subscription_id}/resourceGroups/${var.resource_group}/providers/Microsoft.Compute/images/${var.image_name}-${var.versionToBuild}"
  }
}

I can for instance use SetAttributeRaw like so:

customDataTokens := hclwrite.Tokens{
	{
		Type: hclsyntax.ObjectConsItem
	},
	{
		Type:  hclsyntax.TokenOBrace,
		Bytes: []byte(`{`),
	},
	{
		Type:  hclsyntax.TokenCBrace,
		Bytes: []byte(`}`),
	},
}

body.SetAttributeRaw("custom_data", customDataTokens)

Which will result in:
custom_data = {}

But how should I go on about creating an attribute object which contains multiple attributes with reference values, such as the custom_data example object above?

Unfortunately I’m in a position where using the JSON Configuration Syntax is not possible.

Since as it’s pointed out (in the post I linked to), I’d need to use SetAttributeRaw to meet this use-case with today’s hclwrite, I’m all for that but I have no idea how to do it - an example would be super helpful and really appreciated!

Or, perhaps sine it’s been 2 years since that was posted, perhaps hclwrite now have a helper function for generating this sort of expression? :slight_smile:

2

Hi @aleqsss,

I think for most of what you want to generate here you can do it with these two helper functions:

  • TokensForTraversal to generate an expression like var.subscription_id.
  • TokensForObject to generate an object constructor expression with arbitrary tokens as the keys and values.

That should be enough to generate something like this:

post-processor "manifest" {
  custom_data = {
    subscription        = var.subscription_id
    resource_group      = var.resource_group
    gallery_name        = var.gallery_name
    image_name          = var.image_name
    image_version       = var.versionToBuild
    source_version      = var.info_source_version
    source_branch       = var.info_source_branch
    source_repo         = var.info_source_repo
   }
}

The gallery_artifact_id and managed_artifact_id are trickier because they are complex string template expressions. There isn’t any helper function for building those, so you will need to construct them from raw tokens, but you can at least use the TokensForTraversal helper to generate the tokens between ${ and } in each of your interpolation sequences.

String templates are quite tricky in how they are tokenized, so here are some tips:

  • The raw expression tokens should start with a TokenOQuote and end with a TokenCQuote, both of which having the same bytes []byte{'"'}.
  • The literal string parts will be TokenQuotedLit tokens. Your example doesn’t include any characters that need to be escaped so the bytes will just be literally what you want to generate, but note that if you did need to include something that would need escaping then it must already be escaped in the bytes.
  • The interpolation parts will start with a TokenTemplateInterp whose bytes are literally ${, and end with a TokenTemplateSeqEnd whose bytes are literally }. Between those two markers you can insert the raw result from a call to TokensForTraversal.

Another way you could potentially do this is to write these as calls to the format function instead:

    gallery_artifact_id = format("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/galleries/%s/images/%s/versions/%s", var.subscription_id, var.resource_group, var.gallery_name, var.image_name, var.versionToBuild)

This is of course not how a human would probably write this but should be functionally equivalent from Terraform’s perspective, and can be generated entirely with hclwrite’s helper functions:

  • Use TokensForValue to generate the tokens for the format string itself, passing it as a cty.StringVal.
  • Use TokensForTraversal to generate the subsequent arguments that will be inserted for each of the %s placeholders in the format string.
  • Use TokensForFunctionCall to assemble all of those together into tokens representing the function call syntax.

In all of these cases you’d still pass the resulting Tokens to SetAttributeRaw, but at least you can reduce the need for hand-writing token sequences. Once day I would like to add a higher-level expression-building API to hclwrite which returns Expression values instead of Tokens values, and a way to set an attribute value to an Expression rather than just to its tokens, but unfortunately there is various other work that must take priority over making significant additions to the hclwrite API. I hope you can get something working using the helper functions I mentioned above.

3

Hi @apparentlymart,

Thanks for a good reply, appreciated! This got me going. :+1:

Does something like this look alright?

postProcessor := hclwrite.TokensForObject(
	[]hclwrite.ObjectAttrTokens{
		{
			Name: hclwrite.Tokens{
				{
					Type:  hclsyntax.TokenStringLit,
					Bytes: []byte(`subscription`),
				},
			},
			Value: hclwrite.TokensForTraversal(hcl.Traversal{
				hcl.TraverseRoot{
					Name: "var",
				},
				hcl.TraverseAttr{
					Name: "subscription_id",
				},
			}),
		},
		{
			Name: hclwrite.Tokens{
				{
					Type:  hclsyntax.TokenStringLit,
					Bytes: []byte(`gallery_artifact_id`),
				},
			},
			Value: hclwrite.Tokens{
				{
					Type:  hclsyntax.TokenOQuote,
					Bytes: []byte(`"`),
				},
				{
					Type:  hclsyntax.TokenQuotedLit,
					Bytes: []byte(`/subscriptions/`),
				},
				{
					Type:  hclsyntax.TokenTemplateInterp,
					Bytes: []byte(`${`),
				},
				{
					Bytes: hclwrite.TokensForTraversal(hcl.Traversal{
						hcl.TraverseRoot{
							Name: "var",
						},
						hcl.TraverseAttr{
							Name: "subscription_id",
						},
					}).Bytes(),
				},
				{
					Type:  hclsyntax.TokenTemplateSeqEnd,
					Bytes: []byte(`}`),
				},
				{
					Type:  hclsyntax.TokenCQuote,
					Bytes: []byte(`"`),
				},
			},
		},
	},
)

I’m a bit unsure if I’m doing it correctly. I mean the results visually looks good, but for instance for:

{
	Bytes: hclwrite.TokensForTraversal(hcl.Traversal{
		hcl.TraverseRoot{
			Name: "var",
		},
		hcl.TraverseAttr{
			Name: "subscription_id",
		},
	}).Bytes(),
},

Is this how I should set the TokensForTraversal in the string template, or am I doing it wrong? What should I set as Type:?

Would you have a more elaborate example of your second suggestion, with the format approach? I struggle how to use the parts correctly. For instance, how would I use TokensForTraversal to keep the ${}?

I’d really like to implement the second approach if possible, since it seems to be a lot more straight forward and not having to manually construct all of the tokens.

Thank you for really good input so far!

/Alexander

4

Just read this through once again, and now I get it. You mean that I should literally write out:

gallery_artifact_id = format("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/galleries/%s/images/%s/versions/%s", var.subscription_id, var.resource_group, var.gallery_name, var.image_name, var.versionToBuild)

To the configuration, so that Terraform would use its built in format function when interpreting it, not any format function inside of the Go code.

Sorry for being slow here…

5

For your second suggestion, does this looks ok? Or would you say that there is something that is off? :slightly_smiling_face:

tokenSubscriptionId := hclwrite.TokensForTraversal(hcl.Traversal{
	hcl.TraverseRoot{
		Name: "var",
	},
	hcl.TraverseAttr{
		Name: "subscription_id",
	},
})
tokenResourceGroup := hclwrite.TokensForTraversal(hcl.Traversal{
	hcl.TraverseRoot{
		Name: "var",
	},
	hcl.TraverseAttr{
		Name: "resource_group",
	},
})
tokenGalleryName := hclwrite.TokensForTraversal(hcl.Traversal{
	hcl.TraverseRoot{
		Name: "var",
	},
	hcl.TraverseAttr{
		Name: "gallery_name",
	},
})
tokenImageName := hclwrite.TokensForTraversal(hcl.Traversal{
	hcl.TraverseRoot{
		Name: "var",
	},
	hcl.TraverseAttr{
		Name: "image_name",
	},
})
tokenVersionToBuild := hclwrite.TokensForTraversal(hcl.Traversal{
	hcl.TraverseRoot{
		Name: "var",
	},
	hcl.TraverseAttr{
		Name: "versionToBuild",
	},
})

tokensFormatString := hclwrite.TokensForValue(cty.StringVal("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/galleries/%s/images/%s/versions/%s"))

tokens := hclwrite.TokensForFunctionCall("format", hclwrite.Tokens{
	{
		Bytes: tokensFormatString.Bytes(),
	},
	{
		Type:  hclsyntax.TokenComma,
		Bytes: []byte(","),
	},
	{
		Bytes: tokenSubscriptionId.Bytes(),
	},
	{
		Type:  hclsyntax.TokenComma,
		Bytes: []byte(","),
	},
	{
		Bytes: tokenResourceGroup.Bytes(),
	},
	{
		Type:  hclsyntax.TokenComma,
		Bytes: []byte(","),
	},
	{
		Bytes: tokenGalleryName.Bytes(),
	},
	{
		Type:  hclsyntax.TokenComma,
		Bytes: []byte(","),
	},
	{
		Bytes: tokenImageName.Bytes(),
	},
	{
		Type:  hclsyntax.TokenComma,
		Bytes: []byte(","),
	},
	{
		Bytes: tokenVersionToBuild.Bytes(),
	},
})

body.SetAttributeRaw("gallery_artifact_id", tokens)

Would you say that there is something I could do to optimize it a bit more, or is this as “short” as it gets? Like, do I really need to specify the TokenComma manually and do I use:

{
	Bytes: tokenResourceGroup.Bytes(),
},

correctly? Do I need to specify Type: or should I insert it differently?

Thanks again for your great input!

6

I’m not at a computer right now so I’m relying on my memory but I think TokensForFunctionCall accepts an arbitrary number of Tokens values that each represent one argument and then it’ll insert the commas between them itself.

If that’s true then you can write something like this (untested since I’m on a phone without a Go compiler!):

tokens := hclwrite.TokensForFunctionCall(
    "format",
    tokensFormatString,
    tokenSubscriptionId,
    tokenResourceGroup,
    tokenGalleryName,
    tokenImageName,
    tokenVersionToBuild,
)

One big caveat with this raw token manipulation is that hclwrite will largely just accept whatever you give it even if it wouldn’t be possible to achieve via parsing, so if you hand-write Tokens values with literal type and bytes you can potentially write something that isn’t really correct but hclwrite will try to do something with it anyway. That’s why I recommended relying exclusively on the token-building functions rather than writing out tokens by hand.

Writing invalid combinations of Type and Bytes can appear to work when you’re just requesting the final file bytes anyway, but hclwrite still uses the token types for stuff like adding spaces and indentation between the tokens and so it’s quite easy to generate something that is almost right except that the spacing around the tokens is wrong, or similar.

This problem is what the raw token building helper functions are currently partially solving for and what a full expression-building API will hopefully solve more completely someday.

7

Just tried it out and it worked, thank you very much! :slightly_smiling_face: