Kubernetes_stateful_set not allowing node_affinity to use local persistence volumes

Terraform
1

Hi I am trying to create a statefulset with local persistent volume. So i am trying to use node_affinity to ensure pods getting created in specific nodes but its failing to accept node_selector_term. Kindly help me to fix this of any example to use local persistent volume with node selection will be very much helpful. Thanks.

resource “kubernetes_stateful_set” “es_data” {
metadata {
name = “tf-magicmind-es-datanode”
namespace = “${kubernetes_namespace.magicmind.metadata.0.name}”

labels = {
  component = "elasticsearch"
  role      = "datanode"
  app       = "tf-magicmind-es-datanode"
}

}

spec {
replicas = 1

selector {
  match_labels = {
    app = "tf-magicmind-es-datanode"
  }
}

volume_claim_template {
  metadata {
    name = "tf-magicmind-datanode-pv"
    labels = {
      app = "tf-magicmind-es-datanode"
    }
  }
  spec {
    access_modes       = ["ReadWriteOnce"]
    storage_class_name = "tf-magicmind-datanode"

    resources {
      requests = {
        storage = "5Gi"
      }
    }
  }
}

template {
  metadata {
    labels = {
      component = "elasticsearch"
      role      = "datanode"
      app       = "tf-magicmind-es-datanode"
    }
  }

  spec {


    init_container {
      name    = "init-sysctl"
      image   = "busybox:1.27.2"
      command = ["sysctl", "-w", "vm.max_map_count=262144"]

      security_context {
        privileged = true
      }
    }

    init_container {
      name    = "fix-perm"
      image   = "busybox"
      command = ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
      volume_mount {
        name       = "tf-magicmind-datanode"
        mount_path = "/usr/share/elasticsearch/data"
      }
      security_context {
        privileged = true
      }
    }


    container {
      name  = "es-data"
      image = "docker.elastic.co/elasticsearch/elasticsearch:7.3.0"

      port {
        name           = "transport"
        container_port = 9300
      }

      env {
        name = "NAMESPACE"

        value_from {
          field_ref {
            field_path = "metadata.namespace"
          }
        }
      }

      env {
        name = "NODE_NAME"

        value_from {
          field_ref {
            field_path = "metadata.name"
          }
        }
      }

      env {
        name  = "CLUSTER_NAME"
        value = "tf-magicmind-es"
      }

      env {
        name  = "NODE_MASTER"
        value = "false"
      }

      env {
        name  = "NODE_INGEST"
        value = "false"
      }

      env {
        name  = "HTTP_ENABLE"
        value = "false"
      }

      env {
        name  = "ES_JAVA_OPTS"
        value = "-Xms256m -Xmx256m"
      }

      env {
        name = "PROCESSORS"

        value_from {
          resource_field_ref {
            resource = "limits.cpu"
          }
        }
      }

      resources {
        limits {
          cpu = "1"
        }
      }
    }

    affinity {
      node_affinity {
        preferred_during_scheduling_ignored_during_execution {
          node_selector_term {
            match_expressions  {
                key      = "kubernetes.io/hostname"
                operator = "In"
                values   = [ "${var.node1_name}", "${var.node2_name}" ]
            }
          }
        }
      }
    }
  }
}
service_name = "tf-magicmind-es-datanode"

}
}

2

Hello @baranikannan,

I wasn’t able to find a reference to node_selector_term in the Kubernetes API for affinity but I did see weight and preference. I did get the following node affinity configuration working with weight and preference:

        affinity {
          node_affinity {
            preferred_during_scheduling_ignored_during_execution {
              weight = 1
              preference {
                match_expressions {
                  key      = "kubernetes.io/hostname"
                  operator = "In"
                  values   = [ "${var.node1_name}", "${var.node2_name}" ]
                }
              }
            }
          }
        }
3

Hi Rosemary,

Thank you so much for checking my request. I am following the below code and it works fine. I am not able to track from which git link I found it. I want to achieve running statefulset in specific node since I am using local persistent volume. I will be thankful if you help me on achieving it.

apiVersion: v1
kind: PersistentVolume
metadata:
name: rethinkdb-pv
spec:
capacity:
storage: 15Gi
accessModes:

  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: local-storage
    local:
    path: /mnt/rethinkdb
    nodeAffinity:
    required:
    nodeSelectorTerms:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: rethinkdb-master
spec:
serviceName: rethinkdb-master
replicas: 2
template: # pod template
metadata:
labels:
app: rethinkdb-master
spec:
hostname: rethinkdb-master
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01
- node02
containers:
- name: rethinkdb
image: rethinkdb:2.3.5
# set the canonical address for this node to a custom one
command: [“rethinkdb”]
args:
- --bind
- “all”
- --canonical-address
- “rethinkdb-master:29015”
- --canonical-address
- “$(MY_POD_IP):29015”
volumeMounts:
- name: rdb-local-data
mountPath: /data
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumes:
- name: rdb-local-data
hostPath:
path: /mnt/rethinkdb

apiVersion: v1
kind: Service
metadata:
name: rethinkdb-master
labels:
app: rethinkdb-master
spec:
ports:

  • port: 28015
    name: rdb-api
  • port: 29015
    name: rdb-cluster-api
    selector:
    app: rethinkdb-master
4

Hi @baranikannan,

It would really help to embed code and YAML with markdown syntax! Use three backticks (```) surrounding the sections, for example:

```yaml
apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: rethinkdb-master
```

The StatefulSet you included in the second post in Kubernetes YAML uses the requiredDuringSchedulingIgnoredDuringExecution, which does leverage nodeSelectorTerm and that is why it might be working. In the initial example you included that is in Terraform configuration, the StatefulSet used preferred_during_scheduling_ignored_during_execution. When setting a StatefulSet with preferred, the Kubernetes API looks for weight and preference parameters. This is why Terraform and Kubernetes return an error. If you update your Terraform configuration to use required_during_scheduling_ignored_during_execution instead of preferred, you will be able to use the node_selector_term in your configuration.

I believe your PersistentVolume declaration should be fine when it is converted to Terraform configuration, as it is using required and the node_selector_term.

Hope this helps!