Am I misunderstanding how ignore_fields works in kubectl_manifest Resource?

Terraform Providers Kubernetes
1

Hi all,

I hope you can help me. I’m trying to ignore some fields (i.e. not update them in the Terraform) for the kubectl_manifest resource (see: https://registry.terraform.io/providers/alon-dotan-starkware/kubectl/latest/docs/resources/kubectl_manifest).
I have come across some Documentation here https://github.com/gavinbunney/terraform-provider-kubectl/blob/v1.14.0/docs/resources/kubectl_manifest.md#ignore-manifest-fields

And it states in the Documentation:

For arrays, the syntax is indexed based on the element position. For example, to ignore the caBundle field in the below manifest, would be: webhooks.0.clientConfig.caBundle

The example provided is below:

resource "kubectl_manifest" "test" {
    yaml_body = <<YAML
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
webhooks:
  - clientConfig:
      caBundle: ""
YAML

    ignore_fields = ["webhooks.0.clientConfig.caBundle"]
}

My Manifest Body is here: https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

I would like to ignore_fields for the below line in my Manifest Body:

      - image: "gcr.io/gke-release/nvidia-partition-gpu@sha256:e226275da6c45816959fe43cde907ee9a85c6a2aa8a429418a4cadef8ecdb86a"

This is my Terraform specific to my case:

data "http" "nvidia_driver_installer_manifest" {
  url = "https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml"
}

resource "kubectl_manifest" "nvidia_driver_installer" {
  yaml_body = data.http.nvidia_driver_installer_manifest.response_body
  ignore_fields = ["spec.template.spec.initContainers.1.image"]
}

But for some reason I’m still seeing the output being changed by the Terraform Output.

Have I misunderstood this?

Any help would be greatly appreciated!

Many thanks,
Chris

2

My Terraform Output is below:

Terraform will perform the following actions:

  # kubectl_manifest.nvidia_driver_installer will be updated in-place
  ~ resource "kubectl_manifest" "nvidia_driver_installer" {
        id                      = "/apis/apps/v1/namespaces/kube-system/daemonsets/nvidia-driver-installer"
      + ignore_fields           = [
          + "spec.template.spec.initContainers.2.image",
        ]
        name                    = "nvidia-driver-installer"
      ~ yaml_body               = (sensitive value)
      ~ yaml_body_parsed        = <<-EOT
            apiVersion: apps/v1
            kind: DaemonSet
            metadata:
              labels:
                k8s-app: nvidia-driver-installer
              name: nvidia-driver-installer
              namespace: kube-system
            spec:
              selector:
                matchLabels:
                  k8s-app: nvidia-driver-installer
              template:
                metadata:
                  labels:
                    k8s-app: nvidia-driver-installer
                    name: nvidia-driver-installer
                spec:
                  affinity:
                    nodeAffinity:
                      requiredDuringSchedulingIgnoredDuringExecution:
                        nodeSelectorTerms:
                        - matchExpressions:
                          - key: cloud.google.com/gke-accelerator
                            operator: Exists
                          - key: cloud.google.com/gke-gpu-driver-version
                            operator: DoesNotExist
                  containers:
                  - image: gcr.io/google-containers/pause:2.0
                    name: pause
                  hostNetwork: true
                  hostPID: true
                  initContainers:
                  - env:
                    - name: NVIDIA_INSTALL_DIR_HOST
                      value: /home/kubernetes/bin/nvidia
                    - name: NVIDIA_INSTALL_DIR_CONTAINER
                      value: /usr/local/nvidia
                    - name: VULKAN_ICD_DIR_HOST
                      value: /home/kubernetes/bin/nvidia/vulkan/icd.d
                    - name: VULKAN_ICD_DIR_CONTAINER
                      value: /etc/vulkan/icd.d
                    - name: ROOT_MOUNT_DIR
                      value: /root
                    - name: COS_TOOLS_DIR_HOST
                      value: /var/lib/cos-tools
                    - name: COS_TOOLS_DIR_CONTAINER
                      value: /build/cos-tools
                    image: cos-nvidia-installer:fixed
                    imagePullPolicy: Never
                    name: nvidia-driver-installer
                    resources:
                      requests:
                        cpu: 150m
                    securityContext:
                      privileged: true
                    volumeMounts:
                    - mountPath: /usr/local/nvidia
                      name: nvidia-install-dir-host
                    - mountPath: /etc/vulkan/icd.d
                      name: vulkan-icd-mount
                    - mountPath: /dev
                      name: dev
                    - mountPath: /root
                      name: root-mount
                    - mountPath: /build/cos-tools
                      name: cos-tools
                  - env:
                    - name: LD_LIBRARY_PATH
                      value: /usr/local/nvidia/lib64
          -         image: gcr.io/gke-release/nvidia-partition-gpu@sha256:c54fd003948fac687c2a93a55ea6e4d47ffbd641278a9191e75e822fe72471c2
          +         image: gcr.io/gke-release/nvidia-partition-gpu@sha256:e226275da6c45816959fe43cde907ee9a85c6a2aa8a429418a4cadef8ecdb86a
                    name: partition-gpus
                    resources:
                      requests:
                        cpu: 150m
                    securityContext:
                      privileged: true
                    volumeMounts:
                    - mountPath: /usr/local/nvidia
                      name: nvidia-install-dir-host
                    - mountPath: /dev
                      name: dev
                    - mountPath: /etc/nvidia
                      name: nvidia-config
                  priorityClassName: system-node-critical
                  tolerations:
                  - operator: Exists
                  volumes:
                  - hostPath:
                      path: /dev
                    name: dev
                  - hostPath:
                      path: /home/kubernetes/bin/nvidia/vulkan/icd.d
                    name: vulkan-icd-mount
                  - hostPath:
                      path: /home/kubernetes/bin/nvidia
                    name: nvidia-install-dir-host
                  - hostPath:
                      path: /
                    name: root-mount
                  - hostPath:
                      path: /var/lib/cos-tools
                    name: cos-tools
                  - hostPath:
                      path: /etc/nvidia
                    name: nvidia-config
              updateStrategy:
                type: RollingUpdate
        EOT
        # (13 unchanged attributes hidden)
    }
4

I was also running into something similar, I think I understand now how this works.
The ignore_fields does NOT ignore changes if these changes are in your yaml_body. So if you change the image in the yaml, it will be applied.
However, if the change is not coming from terraform (but i.e. from a kubectl command), it will be ignored here.