# Lists with trailing commas templated by terraform, rejected by AWS **URL:** \ **Category:** Terraform\ **Created:** [February 27, 2020, 3:49pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354 "2020-02-27T15:49:19Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![piyat](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/piyat/32/4977_2.png) [@piyat](https://discuss.hashicorp.com/u/piyat)\ **Post date:** [February 27, 2020, 3:49pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/1 "2020-02-27T15:49:19Z")
Hey! **Background** To give an overview of what I’m doing - I’m working on a project in tf 0.12. The goal is to stand up some databases and data migration infrastructure. I’ve put together a couple of modules to handle that and a calling module to supply input. In my test environment I have 3 RDS instances, 3 DMS replication instances (1 per RDS), and 6 DMS endpoints (3 source, 3 target). What I want to do is allow some database, dev types into the AWS console so they can configure the endpoints, and replication tasks manually and then once happy I can port that into the module. **Issue** I want to create a few sets of aws\_iam\_policy, aws\_iam\_group and aws\_iam\_group\_policy\_attachment resources so that per environment all users in the group will have certain permissions to explicitly defined DMS resources. The IAM policy that is being rendered contains syntax errors, and as far as I can tell terraform is adding a trailing comma to list objects in the policy definition. **Versions** Terraform: 0.12.19 provider.aws: version = “2.46” provider.onepassword: version = “0.5” provider.random: version = “2.2” **What I’m doing** - I’m using outputs/data sources to source 3 tuples of dms ARN’s from a separate child module - one ‘list’ each for source endpoints, target endpoints and repl instances - some example data below. - I’m merging these in a local var into a single list - I’m formatting this var with jsonencode and then attempting to use it as a templating var in a json iam policy template At this point, it looks as though jsonencode has done the trick and I now have a properly formatted string to inject into my template, so I run this: ``` resource "aws_iam_policy" "dms_policy" { name = "tf-dms-permissions-${terraform.workspace}" description = "Policy allowing console users to access dms resources created in terraform workspace." path = "/" policy = templatefile("${path.module}/policies/dms_permissions.json", { dms_resources = jsonencode(local.all_dms_merged) }) } ``` The json policy template saved in ${path.module}/policies/dms\_permissions.json > { > “Version”: “2012-10-17”, > “Sid”: “DMSAllowedOperations”, > “Statement”: [ > { > “Effect”: “Allow”, > “Resource”: [ > ${dms\_resources} > ], > “Action”: [ > “dms:DescribeSchemas”, > “dms:DescribeRefreshSchemasStatus”, > “dms:ModifyReplicationTask”, > “dms:StartReplicationTask”, > “dms:DescribeEventSubscriptions”, > “dms:DescribeEndpointTypes”, > “dms:DescribeEventCategories”, > “dms:StartReplicationTaskAssessment”, > “dms:DescribeOrderableReplicationInstances”, > “dms:ListTagsForResource”, > “dms:DescribeConnections”, > “dms:DescribeReplicationInstances”, > “dms:DeleteReplicationTask”, > “dms:TestConnection”, > “dms:DescribeEndpoints” > ] > } > ] > } **Expected** The rendered policy file is formatted as specified in either the template supplied to the templatefile function or in the data.aws\_iam\_policy\_document renderer. **Actual** The aws iam/CreatePolicy API rejects the rendered policy with: MalformedPolicyDocument 2020/02/27 11:25:38 [DEBUG] aws\_iam\_policy.dms\_policy: apply errored, but we’re indicating that via the Error pointer rather than returning it: Error creating IAM policy example: MalformedPolicyDocument: Syntax errors in policy. The terraform plan looks like this: > # aws\_iam\_policy.dms\_policy will be created > > - resource “aws\_iam\_policy” “dms\_policy” { > - arn = (known after apply) > 2020/02/27 11:25:34 [DEBUG] command: asking for input: “Do you want to perform these actions in workspace "dev"?” > - description = “Policy allowing console users to access dms resources created in terraform workspace.” > - id = (known after apply) > - name = “tf-dms-permissions-dev” > - path = “/” > - policy = jsonencode( > { > + Statement = [ > + { > + Action = [ > + “dms:DescribeSchemas”, > + “dms:DescribeRefreshSchemasStatus”, > + “dms:ModifyReplicationTask”, > + “dms:StartReplicationTask”, > + “dms:DescribeEventSubscriptions”, > + “dms:DescribeEndpointTypes”, > + “dms:DescribeEventCategories”, > + “dms:StartReplicationTaskAssessment”, > + “dms:DescribeOrderableReplicationInstances”, > + “dms:ListTagsForResource”, > + “dms:DescribeConnections”, > + “dms:DescribeReplicationInstances”, > + “dms:DeleteReplicationTask”, > + “dms:TestConnection”, > + “dms:DescribeEndpoints”, > ] > + Effect = “Allow” > + Resource = > + [ > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:rep:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:rep:somevalue”, > “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:rep:somevalue”, > ], > + Sid = “DMSAllowedOperations” > }, > ] > + Version = “2012-10-17” > } > ) > } The problem with this policy it seems is the trailing comma in both the Action list and Resource list. If I drop this policy into policy sim and remove these trailing commas, the json is valid and I’d expect the call to AWS would succeed. If this is a known bug - is there a workaround? ---
**Author:** ![grimm26](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/grimm26/32/1623_2.png) [@grimm26](https://discuss.hashicorp.com/u/grimm26)\ **Post date:** [February 27, 2020, 9:13pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/2 "2020-02-27T21:13:10Z")
You should use [https://www.terraform.io/docs/providers/aws/d/iam\_policy\_document.html](https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html) to create policies ---
**Author:** ![piyat](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/piyat/32/4977_2.png) [@piyat](https://discuss.hashicorp.com/u/piyat)\ **Post date:** [February 28, 2020, 9:05am UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/3 "2020-02-28T09:05:50Z")
Hey Mark - thanks for replying. I did mention above that I tried this too, with the same error. > [@piyat](#): > > - This didn’t work, so I’ve attempted the same using the data source aws\_iam\_policy\_document, but ran into the same error ---
**Author:** ![grimm26](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/grimm26/32/1623_2.png) [@grimm26](https://discuss.hashicorp.com/u/grimm26)\ **Post date:** [February 28, 2020, 3:12pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/4 "2020-02-28T15:12:02Z")
Works fine for me: [https://gist.github.com/grimm26/2a2b1a285edb9d364b906cb607912fd1](https://gist.github.com/grimm26/2a2b1a285edb9d364b906cb607912fd1) I will reiterate that you should not try to build json policy from scratch. You say you tried using aws\_iam\_policy\_document, but you should retry. Don’t use a templatefile to build a policy document. ---
**Author:** ![piyat](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/piyat/32/4977_2.png) [@piyat](https://discuss.hashicorp.com/u/piyat)\ **Post date:** [February 28, 2020, 6:40pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/5 "2020-02-28T18:40:55Z")
Thanks again Mark and cheers for the gist. I agree, when you supply the resource list literally to aws\_iam\_policy\_document, terraform templates it right and I can create the policy. No sweat. The gist I’ve attached illustrates the problem I’m facing: > Maybe I’ve not been clear in how I’ve asked this question - so I really appreciate your help. The bit that I’m not understanding is how to handle supplying the local variable to any variant of terraform resource which outputs json where I need to convert that variable from a tuple to a json compliant list whilst also adhering to the terraform template schema expecting a string. ---
**Author:** ![grimm26](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/grimm26/32/1623_2.png) [@grimm26](https://discuss.hashicorp.com/u/grimm26)\ **Post date:** [February 28, 2020, 7:19pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/6 "2020-02-28T19:19:13Z")
[https://gist.github.com/piyat/840649f821234abefa636afe3932cd10#file-gistfile1-txt-L33-L35](https://gist.github.com/piyat/840649f821234abefa636afe3932cd10#file-gistfile1-txt-L33-L35) this will throw errors because local.a\_test\_var is already a list and you put it in which makes it a list with the first element being a list of strings. Your second test is throwing an error because you are jsonencoding the a\_test\_var variable for some reason. `aws_iam_policy_document` doesn’t expect json - you use it to generate json. It _should_ be: locals { a\_test\_var = [ “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue”, “arn:aws:dms:eu-west-1:xxxxxxxxxxxx:endpoint:somevalue” ] } ``` data "aws_iam_policy_document" "dms" { statement { sid = "DMSAllowedOperations" actions = [ "dms:DescribeSchemas", "dms:DescribeRefreshSchemasStatus", "dms:ModifyReplicationTask", "dms:StartReplicationTask", "dms:DescribeEventSubscriptions", "dms:DescribeEndpointTypes", "dms:DescribeEventCategories", "dms:StartReplicationTaskAssessment", "dms:DescribeOrderableReplicationInstances", "dms:ListTagsForResource", "dms:DescribeConnections", "dms:DescribeReplicationInstances", "dms:DeleteReplicationTask", "dms:TestConnection", "dms:DescribeEndpoints", ] effect = "Allow" resources = local.a_test_var } } ``` ---
**Author:** ![apparentlymart](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/apparentlymart/32/702_2.png) [@apparentlymart](https://discuss.hashicorp.com/u/apparentlymart)\ **Post date:** [March 3, 2020, 11:08pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/7 "2020-03-03T23:08:07Z")
Hi @piyat, Terraform is using its own syntax to render the proposed change to the `policy` argument, after parsing the given value as JSON. You can tell when Terraform is doing this because it includes an extra marker that is intended to resemble a call to [the `jsonencode` function](https://www.terraform.io/docs/configuration/functions/jsonencode.html): ```auto + policy = jsonencode( + { ... and then all of your policy document content ... }, ) ``` The fact that Terraform was able to produce this structural diff of the JSON document tells me that the value _is_ valid JSON internally: Terraform managed to parse it and then format it as a diff in the plan output. The value shown there is not a byte-for-byte representation of the JSON data to be submitted, but rather a pretty-printed representation of the effective data inside that JSON document. (If you were making a change to an existing policy rather than creating a new one, Terraform would use this syntax to show you specifically which part of the data structure had changed.) Therefore it seems that the syntax error mentioned in the error message is not a _JSON_ syntax error, but rather an _IAM policy_ syntax error. That is, an error within one of the values you’ve included in the JSON data, such as invalid ARN syntax or use of an incorrect JSON value type somewhere in the structure. As @grimm26 noted, the `aws_iam_policy_document` data source can help here because it understands the IAM policy schema and so it can (to a certain extent) validate that you’re producing a valid IAM policy data structure _and_ ensure that the result is valid JSON syntax. ---
**Author:** ![piyat](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/piyat/32/4977_2.png) [@piyat](https://discuss.hashicorp.com/u/piyat)\ **Post date:** [March 11, 2020, 9:24am UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/8 "2020-03-11T09:24:01Z")
Hey @grimm26, @apparentlymart. Sorry it’s taken me a while to reply. Thanks ever so much for the explanation. I had confused matters asking a question about two different means of producing Iam policy docs so appreciate you taking the time to help me understand. Indeed I was doing something silly and uneccesarily passing the list to jsonencode in data.aws\_iam\_policy\_document.dms. All works a treat now. Thanks again 😄 ---
**Author:** ![vissie](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/vissie/32/5264_2.png) [@vissie](https://discuss.hashicorp.com/u/vissie)\ **Post date:** [March 11, 2020, 12:02pm UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/9 "2020-03-11T12:02:51Z")
I have exactly the same issue with the following code and I cant figure out how to resolve. ```auto data "aws_iam_policy_document" "this" { statement { effect = "Allow" principals { type = "AWS" identifiers = [ "arn:aws:iam::123456789012:user/test", "arn:aws:iam::012345678901:user/test" ] } actions = [ "s3:GetObject", "s3:ListBucket" ] resources = [ "arn:aws:s3:::${var.bucket_name}", "arn:aws:s3:::${var.bucket_name}/*" ] } } resource "aws_s3_bucket" "terraform_state" { bucket = var.bucket_name policy = data.aws_iam_policy_document.this.json versioning { enabled = var.bucket_versioning } tags = module.tag-label.tags } ``` This produces the following plan ```auto Terraform will perform the following actions: # module.s3-simple.aws_s3_bucket.terraform_state will be updated in-place ~ resource "aws_s3_bucket" "terraform_state" { acl = "private" arn = "arn:aws:s3:::x-terraform-state-playpen" bucket = "x-terraform-state-playpen" bucket_domain_name = "x-terraform-state-playpen.s3.amazonaws.com" bucket_regional_domain_name = "x-terraform-state-playpen.s3.eu-west-1.amazonaws.com" force_destroy = false hosted_zone_id = "Z1BKCTXD74EZPE" id = "x-terraform-state-playpen" ~ policy = jsonencode( ~ { ~ Statement = [ ~ { Action = [ "s3:ListBucket", "s3:GetObject", ] Effect = "Allow" ~ Principal = { ~ AWS = "arn:aws:iam::123456789012:user/other" -> [ + "arn:aws:iam::123456789012:user/test", + "arn:aws:iam::012345678901:user/test", ] } Resource = [ "arn:aws:s3:::x-terraform-state-playpen/*", "arn:aws:s3:::x-terraform-state-playpen", ] Sid = "" }, ] Version = "2012-10-17" } ) region = "eu-west-1" request_payer = "BucketOwner" tags = { "Account" = "x" "Application" = "x" "Name" = "x" "Namespace" = "x" "Stage" = "x" } versioning { enabled = true mfa_delete = false } } Plan: 0 to add, 1 to change, 0 to destroy. ``` and I get the same error ```auto Error: Error putting S3 policy: MalformedPolicy: Invalid principal in policy status code: 400, request id: 72061323F3B9021X, host id: CNTnFYvvVdk1/naGe+RA0jrr0GJFXqgGD+3xLtLOq+oNavZzLm6c4tV3Q5sQNVkzpMLEzGTnn9E= ``` Versions: ```auto Terraform v0.12.23 + provider.aws v2.52.0 ``` ---
**Author:** ![vissie](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/vissie/32/5264_2.png) [@vissie](https://discuss.hashicorp.com/u/vissie)\ **Post date:** [March 12, 2020, 9:14am UTC](https://discuss.hashicorp.com/t/lists-with-trailing-commas-templated-by-terraform-rejected-by-aws/6354/10 "2020-03-12T09:14:17Z")
okay, seems like running it in debug mode it was fine, they seem to check if the iam user actually exists before they allow it to be applied where in the past it wasn’t like that.