Hi,
I have a grpc server, hosted on k8s, I have a busybox service with an upstream to the grpc service but I cant quite get it to work while having the proper tags (I also managed to make it work for a postgres instance with the tcp protocol).
I get the following message
➜ ./grpcurl -plaintext localhost:9097 list
Failed to list services: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination
And I have the following logs in envoy :
[2020-12-08 17:01:22.598][14][debug][rbac] [source/extensions/filters/http/rbac/rbac_filter.cc:68] checking request: requestedServerName: , sourceIP: 192.168.14.35:41224, directRemoteIP: 192.168.14.35:41224, remoteIP: 192.168.14.35:41224,localAddress: 192.168.10.222:20000, ssl: uriSanPeerCertificate: spiffe://69f90c95-0ef2-e5c3-95e4-e9f5761d5012.consul/ns/default/dc/dc3/svc/busybox, dnsSanPeerCertificate: busybox.svc.default.69f90c95.consul, subjectPeerCertificate: CN=busybox.svc.default.69f90c95.consul, headers: ':authority', 'localhost:9097'
':path', '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'
':method', 'POST'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.30.0'
'te', 'trailers'
'x-forwarded-proto', 'http'
'x-request-id', '240642ec-b067-4ee4-b38f-ac612fb16621'
, dynamicMetadata:
[2020-12-08 17:01:22.598][14][debug][rbac] [source/extensions/filters/http/rbac/rbac_filter.cc:113] enforced allowed, matched policy consul-intentions-layer4
[2020-12-08 17:01:22.598][14][debug][router] [source/common/router/router.cc:429] [C135][S15924237861353729489] cluster 'local_app' match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'
[2020-12-08 17:01:22.598][14][debug][router] [source/common/router/router.cc:586] [C135][S15924237861353729489] router decoding headers:
':authority', 'localhost:9097'
':path', '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.30.0'
'te', 'trailers'
'x-forwarded-proto', 'http'
'x-request-id', '240642ec-b067-4ee4-b38f-ac612fb16621'
'x-envoy-expected-rq-timeout-ms', '15000'
[2020-12-08 17:01:22.598][14][debug][pool] [source/common/http/conn_pool_base.cc:71] queueing stream due to no available connections
[2020-12-08 17:01:22.598][14][debug][pool] [source/common/conn_pool/conn_pool_base.cc:104] creating a new connection
[2020-12-08 17:01:22.598][14][debug][client] [source/common/http/codec_client.cc:39] [C136] connecting
[2020-12-08 17:01:22.598][14][debug][connection] [source/common/network/connection_impl.cc:755] [C136] connecting to 127.0.0.1:80
[2020-12-08 17:01:22.598][14][debug][connection] [source/common/network/connection_impl.cc:771] [C136] connection in progress
[2020-12-08 17:01:22.598][14][debug][connection] [source/common/network/connection_impl.cc:611] [C136] connected
[2020-12-08 17:01:22.598][14][debug][client] [source/common/http/codec_client.cc:77] [C136] connected
[2020-12-08 17:01:22.598][14][debug][pool] [source/common/conn_pool/conn_pool_base.cc:205] [C136] attaching to next stream
[2020-12-08 17:01:22.598][14][debug][pool] [source/common/conn_pool/conn_pool_base.cc:126] [C136] creating stream
[2020-12-08 17:01:22.598][14][debug][router] [source/common/router/upstream_request.cc:357] [C135][S15924237861353729489] pool ready
[2020-12-08 17:01:22.598][14][debug][client] [source/common/http/codec_client.cc:132] [C136] protocol error: http/1.1 protocol error: HPE_INVALID_CONSTANT
[2020-12-08 17:01:22.598][14][debug][connection] [source/common/network/connection_impl.cc:106] [C136] closing data_to_write=0 type=1
[2020-12-08 17:01:22.598][14][debug][connection] [source/common/network/connection_impl.cc:202] [C136] closing socket: 1
[2020-12-08 17:01:22.598][14][debug][client] [source/common/http/codec_client.cc:96] [C136] disconnect. resetting 1 pending requests
[2020-12-08 17:01:22.598][14][debug][client] [source/common/http/codec_client.cc:119] [C136] request reset
[2020-12-08 17:01:22.598][14][debug][router] [source/common/router/router.cc:1031] [C135][S15924237861353729489] upstream reset: reset reason: connection termination, transport failure reason:
[2020-12-08 17:01:22.598][14][debug][http] [source/common/http/filter_manager.cc:805] [C135][S15924237861353729489] Sending local reply with details upstream_reset_before_response_started{connection termination}
[2020-12-08 17:01:22.598][14][debug][http] [source/common/http/conn_manager_impl.cc:1435] [C135][S15924237861353729489] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'
'date', 'Tue, 08 Dec 2020 17:01:22 GMT'
'server', 'envoy'
'connection', 'close'
I kinda wonder about protocol error: http/1.1 protocol error: HPE_INVALID_CONSTANT
For reference : here are my server deployment and service :
apiVersion: apps/v1
kind: Deployment
metadata:
name: grpc-server
labels:
app: grpc-server
spec:
replicas: 1
selector:
matchLabels:
app: grpc-server
template:
metadata:
labels:
app: grpc-server
annotations:
consul.hashicorp.com/connect-service-name: grpc-server
consul.hashicorp.com/connect-service-protocol: grpc
consul.hashicorp.com/connect-service-port: "80"
spec:
serviceAccountName: grpc-server
containers:
- name: grpc-server
image: clevasseur/simple_grpc_server:latest
ports:
- containerPort: 80
name: grpc
apiVersion: v1
kind: Service
metadata:
name: grpc-server
spec:
ports:
- name: grpc
port: 80
protocol: TCP
targetPort: grpc
selector:
app: grpc-server
sessionAffinity: None
type: ClusterIP
And here is the busybox pod I use for testing
apiVersion: v1
kind: Pod
metadata:
name: busybox
labels:
app: busybox
annotations:
consul.hashicorp.com/connect-service-upstreams: 'grpc-server:9097:dc3'
spec:
containers:
- image: "envoyproxy/envoy-alpine:v1.14.5"
name : busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
serviceAccountName: busybox
restartPolicy: Always
Any help would be appreciated, thanks.
Marius