Hi there,
We have implemented a custom provider to manage a custom resource over a RESTful API (Specifically, Segment Tracking Plans, using the Segment “Config API” documented here - Config API Overview | Segment Documentation, though I don’t think this detail is really relevant).
We’ve found that our RESTful API isn’t very consistent in the JSON in returns, but these typically are in fields we don’t care about, and so we’ve written a StateFunc (Extending Terraform - Schema Behaviors - Terraform by HashiCorp) to canonicalize the JSON (removing fields we don’t care about, sorting list elements, that sort of thing), so that Terraform isn’t seeing phantom diffs.
This has worked great for us so far, but we are finding that Terraform Maps do not honor StateFuncs, and would appreciate it if you could get this resolved (or at least an explanation as to why this is not supported, as it is not documented anywhere).
We have opened an issue against the relevant Git project:
And also have a PR that resolves this specific issue:
We are aware that we are able to use Diff Suppression functions as a workaround, and this is what we are doing now. However diff suppression has some downsides for our use case, and we’d prefer to be able to use the state functions we’ve implemented consistently throughout our schema.
I’ll try and get the issue and PR into our workstream, but unfortunately can’t promise any prioritization or timeline, unfortunately. We’re a small team with a lot to do, and while I acknowledge how frustrating it is to have PRs or bugs sit and not get engagement, we’ve gotta prioritize for impact and long-term sustainability. I’ll see if I can find some time to take a look at this, though.
I’m very interested in hearing more about this, if you’re feeling generous and want to share more details. It’s really helpful to understand the developer experience and what people need and are struggling with. This is what most of our most popular providers are doing, so I’m super interested in hearing about the shortcomings you’re running into.
Am I correct in assuming that you’re storing this JSON as a string value in the state, and are just canonicalizing it so that it doesn’t diff? I.e., this issue is less about it being JSON, and more about it being a string that is semantically unchanged but comprised of different byte sequences?
I understand completely, and appreciate you having a look at this.
Sure thing.
Our current diff suppression function tests for equivalences between two strings by taking the JSON string values, unmarshal them into structs, canonicalizes/normalizes them, and compare to make sure the structs are identical.
The canonicalizing function is fairly elaborate, but here are the relevant bits for the test case I’m presenting below:
(I have also uploaded the relevant source file for our provider, though what I included above is hopefully sufficient to demonstrate the problem, without being too noisy) resource_tracking_plan.go.txt (18.9 KB)
When we perform a terraform plan, with no anticipated changes, we’ll see the right thing:
...
No changes. Infrastructure is up-to-date.
...
Even though we know there are some JSON differences. This is fine, it is exactly what we want.
However, when we make a small change, in this case changing a description field from “The drug type. (ex: generic)” → “The drug type. (ex: generic) blah”, the terraform plan reports on not only the changes I made, but also changes that has until now been suppressed:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# segment_tracking_plan.plans["tracking_plans/android_consumer_prod"] will be updated in-place
~ resource "segment_tracking_plan" "plans" {
id = "rs_1sr4ZVzwUOWKuNrbY29sOAHyeoZ"
~ rules_event_jsons = {
~ "Checkout Step Completed - v1" = jsonencode(
~ {
~ rules = {
+ $schema = "http://json-schema.org/draft-07/schema#"
~ properties = {
~ properties = {
~ properties = {
~ drug = {
~ properties = {
~ drug_type = {
~ description = "The drug type. (ex: generic)" -> "The drug type. (ex: generic) blah"
# (1 unchanged element hidden)
}
# (5 unchanged elements hidden)
}
+ required = []
# (2 unchanged elements hidden)
}
# (16 unchanged elements hidden)
}
# (2 unchanged elements hidden)
}
# (2 unchanged elements hidden)
}
# (1 unchanged element hidden)
}
# (1 unchanged element hidden)
}
)
# (255 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Note that this actually shows three changes to the resource:
description (as anticipated)
$schema (semantically meaningless for our purpose - the value sometimes changes on the server, as they upgrade their software)
required (also semantically meaningless, our service will sometimes - but not always - omit an empty required list - or replace it with null)
The engineers on our team are generally able to identify the actual changes from “semantically meaningless” changes, but as we roll out our Terraform workspace to a larger (not as technical) audience, it would be helpful if we could eliminate these from the diff report to avoid confusion.
As mentioned in the original issue, using state functions did solve the problem for us, it just unfortunately doesn’t work in TypeMap s.
Let me know if this sufficiently illustrates the problem we are attempting to solve.
Any update on this? It’s hard to write a custom provider that is able to effectively and idiomatically handle sensitive values and keep those values of out of state.
There was a few weeks of radio silence here, and so because my employer has an enterprise license, we were able to get an engineer to send me an e-mail, “trying to understand our use case”.
I responded with what we were trying to do.
And it’s been more radio silence since.
We are a relatively successful start up that IPOed about a year ago, but I’m getting the impression that isn’t enough clout.
If you are in a situation where you have an enterprise license, try using those channels, hopefully two paying customers will get more movement than one.
In the meantime, I’ll try and follow up on my end as well.
