Error from server (InternalError) on kubernetes

Consul
1

Hi,I was trying to deploy learn-consul-kubernetes/layer7-observability on kubernetes,when I apply proxy-defaults.yaml,there is error response,here is error message:
Error from server (InternalError): error when creating “./proxy-defaults.yaml”: Internal error occurred: failed calling webhook “mutate-proxydefaults.consul.hashicorp.com”: Post https://consul-controller-webhook.default.svc:443/mutate-v1alpha1-proxydefaults?timeout=10s: context deadline exceeded.
And here is my consul info:

Screenshot from 2021-05-20 17-17-23
Screenshot from 2021-05-20 17-17-231911×1028 289 KB

plz help me to find the reason and solve it,thx!

2

Hi, where are you deploying Kubernetes?

This sounds similar to an issue like [EKS] Cant deploy new pods after enable connectInject · Issue #519 · hashicorp/consul-helm · GitHub.

3

HI，I deployed kubernetes on my local machine,kubernetes version is v1.19.0 , And I deployed cilium as my CNI.
I tried to deploy hashicups/protgres.yaml,There is the similar error message:

service/postgres created
serviceaccount/postgres created
deployment.apps/postgres created
Error from server (InternalError): error when creating "./postgres.yaml": Internal error occurred: failed calling webhook "mutate-servicedefaults.consul.hashicorp.com": Post https://consul-controller-webhook.default.svc:443/mutate-v1alpha1-servicedefaults?timeout=10s: context deadline exceeded

Here is the pod infomation:

$ kubectl get pods -o wide| grep postg
postgres-7d6f87545-tfw8f                                      1/1     Running   0          3m50s   10.12.1.23    istio   <none>           <none>

$ kubectl describe pod postgres-7d6f87545-tfw8f
Name:         postgres-7d6f87545-tfw8f
Namespace:    default
Priority:     0
Node:         istio/192.168.56.201
Start Time:   Fri, 21 May 2021 08:28:04 +0800
Labels:       app=postgres
              pod-template-hash=7d6f87545
              service=postgres
Annotations:  consul.hashicorp.com/connect-inject: true
              prometheus.io/port: 9102
              prometheus.io/scrape: true
Status:       Running
IP:           10.12.1.23
IPs:
  IP:           10.12.1.23
Controlled By:  ReplicaSet/postgres-7d6f87545
Containers:
  postgres:
    Container ID:  docker://e367e1a2de2b106341add500ba4901ed6a07009692065cfe33d12e7b98ea1c26
    Image:         hashicorpdemoapp/product-api-db:v0.0.11
    Image ID:      docker-pullable://hashicorpdemoapp/product-api-db@sha256:f92941da44ae8847ef312cfe9aeca704e2a2a53826b16fa5f6a5e3f70a926f91
    Port:          5432/TCP
    Host Port:     0/TCP
    Args:
      -c
      listen_addresses=127.0.0.1
    State:          Running
      Started:      Fri, 21 May 2021 08:28:40 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POSTGRES_DB:        products
      POSTGRES_USER:      postgres
      POSTGRES_PASSWORD:  password
    Mounts:
      /var/lib/postgresql/data from pgdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from postgres-token-chbc6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  pgdata:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  postgres-token-chbc6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  postgres-token-chbc6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m29s  default-scheduler  Successfully assigned default/postgres-7d6f87545-tfw8f to istio
  Normal  Pulling    5m29s  kubelet            Pulling image "hashicorpdemoapp/product-api-db:v0.0.11"
  Normal  Pulled     4m55s  kubelet            Successfully pulled image "hashicorpdemoapp/product-api-db:v0.0.11"
  Normal  Created    4m54s  kubelet            Created container postgres
  Normal  Started    4m54s  kubelet            Started container postgres

Here is inject-webhook logs:

 kubectl logs consul-connect-injector-webhook-deployment-6fc4bf8968-th5zt
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
2021/05/20 07:19:40 http: TLS handshake error from 10.12.1.112:41250: No certificate available.
2021-05-20T07:19:41.154Z [ERROR] cleanupResource: unable to get Consul services: error="Get "http://192.168.56.201:8500/v1/catalog/services": dial tcp 192.168.56.201:8500: connect: connection refused"
2021-05-20T07:24:41.178Z [ERROR] cleanupResource: unable to get Consul services: error="Unexpected response code: 500 (No known Consul servers)"
Error updating MutatingWebhookConfiguration: etcdserver: request timed out
Error updating MutatingWebhookConfiguration: etcdserver: request timed out

I got all information,But I can’t find the Ip:10.12.1.112

NAME                                                              READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
pod/cc-door-mgr-75754b8c54-qq2tg                                  1/1     Running   0          23d     10.12.2.230   paas    <none>           <none>
pod/coffee-5f56ff9788-9mmqk                                       1/1     Running   0          19d     10.12.0.228   ms      <none>           <none>
pod/coffee-5f56ff9788-nmv24                                       1/1     Running   0          19d     10.12.1.213   istio   <none>           <none>
pod/consul-connect-injector-webhook-deployment-6fc4bf8968-th5zt   1/1     Running   0          17h     10.12.1.78    istio   <none>           <none>
pod/consul-controller-77548c95bd-lp7zw                            1/1     Running   2          17h     10.12.2.95    paas    <none>           <none>
pod/consul-d559p                                                  1/1     Running   0          17h     10.12.1.158   istio   <none>           <none>
pod/consul-jhhsd                                                  1/1     Running   0          17h     10.12.2.226   paas    <none>           <none>
pod/consul-m8kql                                                  1/1     Running   0          17h     10.12.0.108   ms      <none>           <none>
pod/consul-server-0                                               1/1     Running   0          17h     10.12.1.68    istio   <none>           <none>
pod/consul-server-1                                               1/1     Running   0          17h     10.12.2.143   paas    <none>           <none>
pod/consul-server-2                                               1/1     Running   0          17h     10.12.0.65    ms      <none>           <none>
pod/consul-webhook-cert-manager-9d6dbf8f5-6vhm4                   1/1     Running   0          17h     10.12.1.51    istio   <none>           <none>
pod/deathstar-85fdf6498b-ftzj2                                    1/1     Running   5          89d     10.12.0.37    ms      <none>           <none>
pod/deathstar-85fdf6498b-v2pqp                                    1/1     Running   0          23d     10.12.2.9     paas    <none>           <none>
pod/empire-backup-7bb56f5d65-h4dzc                                1/1     Running   24         71d     10.12.0.248   ms      <none>           <none>
pod/empire-hq-5ccff6b587-r2xvx                                    1/1     Running   26         80d     10.12.0.151   ms      <none>           <none>
pod/empire-outpost-8888-59cfb7787f-82bkx                          1/1     Running   6          23d     10.12.2.120   paas    <none>           <none>
pod/empire-outpost-9999-cfccfd579-mckpp                           1/1     Running   6          23d     10.12.2.76    paas    <none>           <none>
pod/gsmessage-6fccc99599-pwvcc                                    1/1     Running   0          2d17h   10.12.1.84    istio   <none>           <none>
pod/kafka-broker-749744dc55-6vp8n                                 1/1     Running   0          23d     10.12.2.103   paas    <none>           <none>
pod/mongo-68d579f669-qs98q                                        1/1     Running   1          56d     10.12.0.209   ms      <none>           <none>
pod/mydemo01pod-79d76f8586-tcf6l                                  1/1     Running   0          16d     10.12.1.83    istio   <none>           <none>
pod/mydemo02pod-868c5977d6-cxfgh                                  1/1     Running   0          16d     10.12.2.225   paas    <none>           <none>
pod/mydemo02pod-868c5977d6-gpf87                                  1/1     Running   0          16d     10.12.1.224   istio   <none>           <none>
pod/mydemo02pod-868c5977d6-lllwr                                  1/1     Running   0          16d     10.12.0.233   ms      <none>           <none>
pod/postgres-7d6f87545-tfw8f                                      1/1     Running   0          18m     10.12.1.23    istio   <none>           <none>
pod/tea-69c99ff568-6sqj2                                          1/1     Running   0          19d     10.12.1.104   istio   <none>           <none>
pod/tea-69c99ff568-f77hh                                          1/1     Running   0          19d     10.12.1.82    istio   <none>           <none>
pod/tea-69c99ff568-qxqv7                                          1/1     Running   0          19d     10.12.2.66    paas    <none>           <none>
pod/zookeeper-5585ff4557-r4wvt                                    1/1     Running   5          80d     10.12.0.68    ms      <none>           <none>

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                   AGE     SELECTOR
service/cc-door-server                ClusterIP   10.32.196.243   <none>        50051/TCP                                                                 76d     app=cc-door-mgr
service/coffee-svc                    ClusterIP   10.32.27.86     <none>        80/TCP                                                                    19d     app=coffee
service/consul-connect-injector-svc   ClusterIP   10.32.112.48    <none>        443/TCP                                                                   17h     app=consul,component=connect-injector,release=consul
service/consul-controller-webhook     ClusterIP   10.32.95.191    <none>        443/TCP                                                                   17h     app=consul,chart=consul-helm,component=controller,heritage=Helm,release=consul
service/consul-dns                    ClusterIP   10.32.242.113   <none>        53/TCP,53/UDP                                                             17h     app=consul,hasDNS=true,release=consul
service/consul-server                 ClusterIP   None            <none>        8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   17h     app=consul,component=server,release=consul
service/consul-ui                     ClusterIP   10.32.41.108    <none>        80/TCP                                                                    17h     app=consul,component=server,release=consul
service/deathstar                     ClusterIP   10.32.79.69     <none>        80/TCP                                                                    89d     class=deathstar,org=empire
service/gsmessage                     ClusterIP   10.32.248.23    <none>        8080/TCP                                                                  2d17h   app=gsmessage
service/kafka-service                 ClusterIP   None            <none>        9092/TCP                                                                  80d     app=kafka
service/kubernetes                    ClusterIP   10.32.0.1       <none>        443/TCP                                                                   101d    <none>
service/mongo                         ClusterIP   10.32.66.13     <none>        27017/TCP                                                                 71d     app.kubernetes.io/component=backend,app.kubernetes.io/name=mongo
service/mydemo01svc                   ClusterIP   10.32.131.79    <none>        80/TCP                                                                    16d     app=mydemo01
service/mydemo02svc                   ClusterIP   10.32.77.78     <none>        8080/TCP                                                                  16d     app=mydemo02
service/postgres                      ClusterIP   10.32.171.121   <none>        5432/TCP                                                                  18m     app=postgres
service/tea-svc                       ClusterIP   10.32.179.3     <none>        80/TCP                                                                    19d     app=tea
service/zook                          ClusterIP   10.32.158.146   <none>        2181/TCP                                                                  80d     app=zook

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE   CONTAINERS   IMAGES                   SELECTOR
daemonset.apps/consul   3         3         3       3            3           <none>          17h   consul       hashicorp/consul:1.9.4   app=consul,chart=consul-helm,component=client,hasDNS=true,release=consul

NAME                                                         READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS             IMAGES                                    SELECTOR
deployment.apps/cc-door-mgr                                  1/1     1            1           76d     cc-door-mgr            docker.io/cilium/cc-grpc-demo:v3.0        app=cc-door-mgr
deployment.apps/coffee                                       2/2     2            2           19d     coffee                 nginxdemos/nginx-hello:plain-text         app=coffee
deployment.apps/consul-connect-injector-webhook-deployment   1/1     1            1           17h     sidecar-injector       hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=connect-injector,release=consul
deployment.apps/consul-controller                            1/1     1            1           17h     controller             hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=controller,heritage=Helm,release=consul
deployment.apps/consul-webhook-cert-manager                  1/1     1            1           17h     webhook-cert-manager   hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=webhook-cert-manager,heritage=Helm,release=consul
deployment.apps/deathstar                                    2/2     2            2           89d     deathstar              docker.io/cilium/starwars                 class=deathstar,org=empire
deployment.apps/empire-backup                                1/1     1            1           80d     empire-backup          docker.io/cilium/kafkaclient              app=empire-backup
deployment.apps/empire-hq                                    1/1     1            1           80d     empire-hq              docker.io/cilium/kafkaclient              app=empire-hq
deployment.apps/empire-outpost-8888                          1/1     1            1           80d     empire-outpost-8888    docker.io/cilium/kafkaclient              app=empire-outpost,outpostid=8888
deployment.apps/empire-outpost-9999                          1/1     1            1           80d     empire-outpost-9999    docker.io/cilium/kafkaclient              app=empire-outpost,outpostid=9999
deployment.apps/gsmessage                                    1/1     1            1           2d17h   gsmessage              gsmessagingstompwebsocket:latest          app=gsmessage
deployment.apps/kafka-broker                                 1/1     1            1           80d     kafka                  docker.io/wurstmeister/kafka:1.1.0        app=kafka
deployment.apps/mongo                                        1/1     1            1           71d     mongo                  mongo:4.2                                 app.kubernetes.io/component=backend,app.kubernetes.io/name=mongo
deployment.apps/mydemo01pod                                  1/1     1            1           16d     myapp                  ikubernetes/myapp:v2                      app=mydemo01
deployment.apps/mydemo02pod                                  3/3     3            3           16d     mytomcat               tomcat:9                                  app=mydemo02
deployment.apps/postgres                                     1/1     1            1           18m     postgres               hashicorpdemoapp/product-api-db:v0.0.11   app=postgres,service=postgres
deployment.apps/tea                                          3/3     3            3           19d     tea                    nginxdemos/nginx-hello:plain-text         app=tea
deployment.apps/zookeeper                                    1/1     1            1           80d     zookeeper              docker.io/digitalwonderland/zookeeper     app=zook

NAME                                                                    DESIRED   CURRENT   READY   AGE     CONTAINERS             IMAGES                                    SELECTOR
replicaset.apps/cc-door-mgr-75754b8c54                                  1         1         1       76d     cc-door-mgr            docker.io/cilium/cc-grpc-demo:v3.0        app=cc-door-mgr,pod-template-hash=75754b8c54
replicaset.apps/coffee-5f56ff9788                                       2         2         2       19d     coffee                 nginxdemos/nginx-hello:plain-text         app=coffee,pod-template-hash=5f56ff9788
replicaset.apps/consul-connect-injector-webhook-deployment-6fc4bf8968   1         1         1       17h     sidecar-injector       hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=connect-injector,pod-template-hash=6fc4bf8968,release=consul
replicaset.apps/consul-controller-77548c95bd                            1         1         1       17h     controller             hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=controller,heritage=Helm,pod-template-hash=77548c95bd,release=consul
replicaset.apps/consul-webhook-cert-manager-9d6dbf8f5                   1         1         1       17h     webhook-cert-manager   hashicorp/consul-k8s:0.25.0               app=consul,chart=consul-helm,component=webhook-cert-manager,heritage=Helm,pod-template-hash=9d6dbf8f5,release=consul
replicaset.apps/deathstar-85fdf6498b                                    2         2         2       89d     deathstar              docker.io/cilium/starwars                 class=deathstar,org=empire,pod-template-hash=85fdf6498b
replicaset.apps/empire-backup-7bb56f5d65                                1         1         1       80d     empire-backup          docker.io/cilium/kafkaclient              app=empire-backup,pod-template-hash=7bb56f5d65
replicaset.apps/empire-hq-5ccff6b587                                    1         1         1       80d     empire-hq              docker.io/cilium/kafkaclient              app=empire-hq,pod-template-hash=5ccff6b587
replicaset.apps/empire-outpost-8888-59cfb7787f                          1         1         1       80d     empire-outpost-8888    docker.io/cilium/kafkaclient              app=empire-outpost,outpostid=8888,pod-template-hash=59cfb7787f
replicaset.apps/empire-outpost-9999-cfccfd579                           1         1         1       80d     empire-outpost-9999    docker.io/cilium/kafkaclient              app=empire-outpost,outpostid=9999,pod-template-hash=cfccfd579
replicaset.apps/gsmessage-6fccc99599                                    1         1         1       2d17h   gsmessage              gsmessagingstompwebsocket:latest          app=gsmessage,pod-template-hash=6fccc99599
replicaset.apps/kafka-broker-749744dc55                                 1         1         1       80d     kafka                  docker.io/wurstmeister/kafka:1.1.0        app=kafka,pod-template-hash=749744dc55
replicaset.apps/mongo-68d579f669                                        1         1         1       71d     mongo                  mongo:4.2                                 app.kubernetes.io/component=backend,app.kubernetes.io/name=mongo,pod-template-hash=68d579f669
replicaset.apps/mydemo01pod-79d76f8586                                  1         1         1       16d     myapp                  ikubernetes/myapp:v2                      app=mydemo01,pod-template-hash=79d76f8586
replicaset.apps/mydemo02pod-868c5977d6                                  3         3         3       16d     mytomcat               tomcat:9                                  app=mydemo02,pod-template-hash=868c5977d6
replicaset.apps/postgres-7d6f87545                                      1         1         1       18m     postgres               hashicorpdemoapp/product-api-db:v0.0.11   app=postgres,pod-template-hash=7d6f87545,service=postgres
replicaset.apps/tea-69c99ff568                                          3         3         3       19d     tea                    nginxdemos/nginx-hello:plain-text         app=tea,pod-template-hash=69c99ff568
replicaset.apps/zookeeper-5585ff4557                                    1         1         1       80d     zookeeper              docker.io/digitalwonderland/zookeeper     app=zook,pod-template-hash=5585ff4557

NAME                             READY   AGE   CONTAINERS   IMAGES
statefulset.apps/consul-server   3/3     17h   consul       hashicorp/consul:1.9.4

The sidecar did not startup

4

Hmm, I think it’s a similar error to [EKS] Cant deploy new pods after enable connectInject · Issue #519 · hashicorp/consul-helm · GitHub where the kube API server can’t reach the pod IPs.

Are you on the latest version of cilium?

5

Here is my cilium info:

kubectl exec cilium-8nchc -n kube-system -- cilium version 
Client: 1.9.90 074407dab 2020-12-11T13:51:06+01:00 go version go1.15.6 linux/amd64
Daemon: 1.9.90 074407dab 2020-12-11T13:51:06+01:00 go version go1.15.6 linux/amd64

In addition to the above suggestions([EKS] Cant deploy new pods after enable connectInject · Issue #519 · hashicorp/consul-helm · GitHub), is there any other way to solve the above problems?

6

Can you run sonobuoy and confirm that your Kubernetes cluster is working as expected: GitHub - vmware-tanzu/sonobuoy: Sonobuoy is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of Kubernetes conformance tests and other plugins in an accessible and non-destructive manner.. I think this may not be a Consul issue.

If you paste the sonobuoy results here and your cluster is conformant we’ll look into reproducing.