Consul helm chart sometimes fails to install CRDs

Consul
1

Hi all,

I noticed that sometimes when I install the consul helm chart it fails to register/install the CRDs. The problem is almost always resolved when reinstalling the chart again.

consul-helm version: 0.36.0
consul version: 1.10.3
kubernetes version: 1.20.7
My consul helm config 
global:
  domain: consul
  name: consul
  gossipEncryption:
    autoGenerate: true
  tls:
    enabled: true
    enableAutoEncrypt: true
  acls:
    manageSystemACLs: true
server:
  replicas: 3
  bootstrapExpect: 3
  extraConfig: |
    {
      "ui_config": {
        "content_path": "/consul"
      }
    }
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "200Mi"
      cpu: "200m"
ui:
  enabled: true
  service:
    type: "ClusterIP"
  metrics:
    enabled: true
    provider: "prometheus"
    baseURL: http://prometheus-prometheus:9090
client:
  tolerations: |
    - operator: Exists
  extraConfig: |
    {"advertise_reconnect_timeout": "15m"}
connectInject:
  enabled: true
  transparentProxy:
    defaultEnabled: true
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "200Mi"
      cpu: "200m"
  sidecarProxy:
    resources:
      requests:
        memory: "100Mi"
        cpu: "100m"
      limits:
        memory: "200Mi"
        cpu: "200m"
controller:
  enabled: true

Let me know if you need further information/details on the issue or the configuration I use.

Best regards,
Nico

2

What exactly goes wrong? Like what errors etc are there?

3

I did a bit more testing. I think the CRD issue only happens if I uninstall the consul helm chart and my custom chart (which uses consul CRDs) at the same time. After both charts are uninstalled the existing consul CRDS are not properly cleaned up and then when reinstall first the consul chart and then my chart I get following errors:

when running helm install on my chart

Error: unable to build kubernetes objects from release manifest: [unable to recognize “”: no matches for kind “ServiceDefaults” in version “consul.hashicorp.com/v1alpha1”, unable to recognize “”: no matches for kind “ServiceIntentions” in version “consul.hashicorp.com/v1alpha1”]

errors logs in consul controller

k8s.io/client-go/tools/cache.(*Reflector).Run.func1
/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156
k8s.io/client-go/tools/cache.(*Reflector).Run
/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1
/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73
2021-11-10T08:51:59.677Z ERROR pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1alpha1.ServiceDefaults: failed to list *v1alpha1.ServiceDefaults: the server could not find the requested resource (get servicedefaults.consul.hashicorp.com)

I hope that helps somehow, I did not find any errors in other consul components

4

Ahh yes uninstalling is difficult. Because if you have resources created from the CRD then those can’t be deleted because there’s a finalizer. But the finalizer can only be removed from our controller. If the controller is uninstalled the the resource is never deleted and then the CRD can’t be deleted.

Uninstall CRDs on helm uninstall · Issue #646 · hashicorp/consul-helm · GitHub shows a way to delete them.

We’re going to add this to our CLI but just haven’t yet.

1 Like