our team use Terraform Splunk provider for managing saved searches.
Usually terraform plan detects changes within HEREDOC section in for all alerts and XYZ unchanged attributes hidden after several days from previous successful terraform apply without any human intervention.
All alerts use Terraform module and theirs syntax is very similar, something like:
module "GuardDuty-Uncommonevents_old" {
source = "git::ssh://fake-repo"
alert_name = "GuardDuty-Uncommonevents_old"
search = <<-EOT
```!!! This alert is managed via Terraform, DON'T modify it directly in Splunk !!! https://fake-repo```
`GuardDuty_old` usually very long search
| rename detail.accountId as XYZ
| eval "Account ID" = XYZ
| lookup jamf_org_accounts.csv XYZ
| stats values(region) as Region
EOT
action_email_message_alert = <<-EOT
'$name$' has occured.
Please check the event below and follow: https://XYZ
Event type: $result.EventType$
Description: $result.Description$
Environment: $result.Environment$
Region: $result.Region$
Account name: $result.Account name$
Account ID: $result.Account ID$
----------------------
$result._raw$
EOT
description = "This alert is managed via Terraform."
}
Running terraform plan after several days, it detects changes within HEREDOC sections - lines:
~ action_email_message_alert = <<-EOT ...
~ search = <<-EOT ....
Needless to say, there are no visible discepancy between the deployed and proposed code. I also downloaded remote state file before and after applying proposed changes, but the remote state does not change. On top of that all searches are formatted with terraform fmt.
I wonder is it a bug within Splunk provider or terraform itself? Does anyone have an idea how to debug it please? Or do you think the discrepancy is caused by some Splunk backend service?
You’ve hidden the actual changed content, which would be most relevant to see!
That’s as expected. Terraform is always applying what is written in your configuration. On the other hand, if you were to then apply a -refresh-only run, that would pull the modified values into the Terraform state.
The described behaviour is consistent with Splunk performing some transformation - such as changing whitespace - which leads to different bytes being subsequently returned.
Legit point, here is an example of another saved search (alert):
module "CloudTrail-AWSRootConsoleLogin" {
source = "git::ssh://XYZ"
alert_name = "CloudTrail-AWSRootConsoleLogin"
search = <<-EOT
```!!! This alert is managed via Terraform, DON'T modify it directly in Splunk !!! https://XYZ```
index=secops_cloudtrail eventName="ConsoleLogin" userIdentity.type="Root"
| eval "Account ID" = aws_account_id
| lookup XYZ
| stats values(eventName) as "Event name" XYZ
EOT
action_email_to = "fakeaddress@fake.opsgenie.net"
action_email_message_alert = <<-EOT
The alert condition for '$name$' was triggered.
Please review results below and follow the playbook
https://fakedomain
Event name: $result.Event name$
Action: $result.Action$
Request group: $result.Request group$
Source: $result.Source$
Agent: $result.Agent$
User: $result.User$
Role: $result.Role$
Environment: $result.Environment$
Region: $result.Region$
Account name: $result.Account name$
Account ID: $result.Account ID$
----------------------
$result._raw$
EOT
description = "This alert is managed via Terraform"
}
Running $ terraform plan -target=module.CloudTrail-AWSRootConsoleLogin, the output looks like:
Acquiring state lock. This may take a few moments...
module.CloudTrail-AWSRootConsoleLogin.splunk_saved_searches.saved_search: Refreshing state... [id=CloudTrail-AWSRootConsoleLogin]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.CloudTrail-AWSRootConsoleLogin.splunk_saved_searches.saved_search will be updated in-place
~ resource "splunk_saved_searches" "saved_search" {
~ action_email_message_alert = <<-EOT
The alert condition for '$name$' was triggered.
Please review results below and follow the playbook
https://fakedomain
Event name: $result.Event name$
Action: $result.Action$
Request group: $result.Request group$
Source: $result.Source$
Agent: $result.Agent$
User: $result.User$
Role: $result.Role$
Environment: $result.Environment$
Region: $result.Region$
Account name: $result.Account name$
Account ID: $result.Account ID$
----------------------
$result._raw$
EOT
id = "CloudTrail-AWSRootConsoleLogin"
name = "CloudTrail-AWSRootConsoleLogin"
~ search = <<-EOT
```!!! This alert is managed via Terraform, DON'T modify it directly in Splunk !!! https://XYZ```
index=secops_cloudtrail eventName="ConsoleLogin" userIdentity.type="Root"
| eval "Account ID" = aws_account_id
| lookup XYZ
| stats values(eventName) as "Event name" XYZ
EOT
# (112 unchanged attributes hidden)
# (1 unchanged block hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an
│ error message.
Alright, I haven’t known about this. And running apply with -refresh-only it says “Note: Objects have changed outside of Terraform”. But what are the use-cases to safely us this option please?
Yeah, that’s I’m afraid of as I don’t know what would be a workaround it. Maybe always run TF apply with -refresh-only ?
Maybe always run TF apply with