Etcdv3 backend, cannot see data in etcd

Hi All,

I have been trying to migrate my state to a backend using an etcdv3 cluster but not sure where I am going wrong and would appreciate some guidance, I did some googling, github and SO searching and have not come up with much.

I am getting the following message about wrapping etcdv3 into a local backend, it appears as if etcdv3 is being used but then querying the etcd cluster yields no keys.

[TRACE] Meta.Backend: backend *etcd.Backend does not support operations, so wrapping it in a local backend

Backend Config:

terraform {
  backend "etcdv3" {
    endpoints = ["10.10.23.54:2379", "10.10.23.55:2379", "10.10.23.56:2379"]
    #endpoints = ["10.10.23.60:23790"]
    lock      = true
    prefix    = "terraform-state/"
    cacert_path = "/pki/etcd_cluster/etcd.pem"
    cert_path = "/pki/etcd_cluster/etcd.pem"
    key_path = "/pki/etcd_cluster/etcd-key.pem"
  }
}

Trace of backend init:

2019/07/18 16:44:04 [INFO] Terraform version: 0.12.4
2019/07/18 16:44:04 [INFO] Go runtime version: go1.12.4
2019/07/18 16:44:04 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}
2019/07/18 16:44:04 [DEBUG] Attempting to open CLI config file: ~/.terraformrc
2019/07/18 16:44:04 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/07/18 16:44:04 [INFO] CLI command args: []string{"init"}
2019/07/18 16:44:04 [TRACE] Meta.Backend: built configuration for "etcdv3" backend with hash value 4014294268
2019/07/18 16:44:04 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:04 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:04 [TRACE] Meta.Backend: working directory was previously initialized for "etcdv3" backend
2019/07/18 16:44:04 [TRACE] backendConfigNeedsMigration: configuration values have changed, so migration is required
2019/07/18 16:44:04 [TRACE] Meta.Backend: backend configuration has changed (from type "etcdv3" to type "etcdv3")
2019/07/18 16:44:04 [WARN] backend config has changed since last init
2019/07/18 16:44:04 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:04 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:04 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:04 [TRACE] backendMigrateState: need to migrate from "etcdv3" to "etcdv3" backend config
2019/07/18 16:44:04 [TRACE] backendMigrateState: migrating all named workspaces
2019/07/18 16:44:04 [DEBUG] command: asking for input: "Do you want to migrate all workspaces to \"etcdv3\"?"
2019/07/18 16:44:06 [TRACE] backendMigrateState: migrating "default" workspace to "default" workspace
2019/07/18 16:44:07 [TRACE] backendMigrateState: both source and destination workspaces have equal state with lineage "f79277a1-73c3-3ac9-c337-19e9367dba06", so no migration is needed
2019/07/18 16:44:07 [TRACE] backendMigrateState: migrating "default" workspace to "default" workspace
2019/07/18 16:44:08 [TRACE] backendMigrateState: both source and destination workspaces have equal state with lineage "f79277a1-73c3-3ac9-c337-19e9367dba06", so no migration is needed
2019/07/18 16:44:08 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:08 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:08 [TRACE] Preserving existing state lineage "7c2bedb7-8dc4-814a-0d9c-b91f0694d7b6"
2019/07/18 16:44:09 [TRACE] Meta.Backend: instantiated backend of type *etcd.Backend
2019/07/18 16:44:09 [DEBUG] checking for provider in "."
2019/07/18 16:44:09 [DEBUG] checking for provider in "/usr/local/bin"
2019/07/18 16:44:09 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/07/18 16:44:09 [DEBUG] found provider "terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] found valid plugin: "vsphere", "1.12.0", "/dev_code/pe-etcd-ha/pe-terraform-etcd-lab-02/.terraform/plugins/darwin_amd64/terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] checking for provisioner in "."
2019/07/18 16:44:09 [DEBUG] checking for provisioner in "/usr/local/bin"
2019/07/18 16:44:09 [DEBUG] checking for provisioner in ".terraform/plugins/darwin_amd64"
2019/07/18 16:44:09 [TRACE] Meta.Backend: backend *etcd.Backend does not support operations, so wrapping it in a local backend
2019/07/18 16:44:09 [DEBUG] checking for provider in "."
2019/07/18 16:44:09 [DEBUG] checking for provider in "/usr/local/bin"
2019/07/18 16:44:09 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/07/18 16:44:09 [DEBUG] found provider "terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] found valid plugin: "vsphere", "1.12.0", "/dev_code/pe-etcd-ha/pe-terraform-etcd-lab-02/.terraform/plugins/darwin_amd64/terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] plugin requirements: "vsphere"=""
2019/07/18 16:44:09 [DEBUG] checking for provider in "."
2019/07/18 16:44:09 [DEBUG] checking for provider in "/usr/local/bin"
2019/07/18 16:44:09 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/07/18 16:44:09 [DEBUG] found provider "terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] found valid plugin: "vsphere", "1.12.0", "/dev_code/pe-etcd-ha/pe-terraform-etcd-lab-02/.terraform/plugins/darwin_amd64/terraform-provider-vsphere_v1.12.0_x4"
2019/07/18 16:44:09 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/07/18 16:44:09 [DEBUG] found provider "terraform-provider-vsphere_v1.12.0_x4"

Query etcd node post terraform run:

[root@etcd01-prod-ctn ~]# curl --key /etc/etcd/tls/etcd-key.pem --cert /etc/etcd/tls/etcd.pem -L https://127.0.0.1:2379/v2/keys/
{"action":"get","node":{"dir":true}}
[root@etcd01-prod-ctn ~]#