Waypoint + kubernetes + runner + consul service mesh

I’m trying to build and deploy app to Kubernetes with consul service mesh. But the problem is waypoint is creating service after deployment is up (as far as I understand ) and consul sidecar is looking for the service to complete registration so it’s kind of stuck.
I tried to switch to Kubernetes-Apply but since there’s no kubectl on kubernetes runner I can’t use this as a workaround.

Anyone was able to make all those things work together ?

So if anyone is trying to do the same here’s how I’ve made it work so far. Not sure if that’s the right way and I’d love some feedback on that.

  1. Used kubectl-apply for “deploy” because I can create Kubernetes service with proper annotations.
  2. Since I’m using remote runner current on-demand runner doesn’t have kubectl binary.
  3. I’ve created a new on-demand-runner container image. Downloaded kubectl binary. “Chmoded” it to 755. Edited this dockefile waypoint/Dockerfile at main · hashicorp/waypoint · GitHub and added kubectl to /kaniko/bin after line 44.
  4. Pushed it to my dockerhub and updated runner profile to use my on demand runner image.

I’ve got my workloads deployed and running in a namespace with Consul Service Mesh enabled.

I’m facing a different challenge now - it seems that waypoint entry point can’t connect to waypoint server.
Here’s what I’ve checked:

  1. I have all the environment variable inside the pod that are needed by the entry point to connect
  2. If I “kubectl exec” to the container and start another waypoint entrypoint with /bin/bash it successfully starts and connects to waypoint server
  3. If I exclude outgoing traffic to the waypoint server it works

What I think is the problem:

  1. My application container with waypoint entry point starts before envoy so connection to waypoint server fails
  2. Waypoint entrypoint doesn’t retry enough times in the background after starting the application, when envoy is up and running.

Did anyone made it work successfully with functional entrypoint capabilities and Consul Service Mesh with Transparent Proxy?

Hi lev2,

Could we see the kubernetes yaml that you used to get things kicked off correctly? We want to see the order so we can figure out how to update the kubernetes plugin to handle this case correctly.

As for the entrypoint, can you provide some logs of it attempting to connect? Is it trying to connect through the service mesh?

Hi,

This is my deployment yaml template that is used by kubectl-apply

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: hashinetes-ci-deployment
  labels:
    app: hashinetes-ci 
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hashinetes-ci
  template:
    metadata:
      annotations:
         "consul.hashicorp.com/connect-inject": "true"
      labels:
        app: hashinetes-ci
    spec:
      serviceAccountName: hashinetes-ci
      containers:
      - name: hashinetes-ci
        image: ${artifact.image}:${artifact.tag}
        ports:
        - containerPort: 8080
        env:
          %{ for k,v in entrypoint.env }
          - name: ${k}
            value: "${v}"
          %{ endfor }
          - name: PORT
            value: "8080"
          - name: WAYPOINT_LOG_LEVEL
            value: "TRACE"

Here are the logs of the pod (application container )

Defaulted container "hashinetes-ci" out of: hashinetes-ci, envoy-sidecar, copy-consul-bin (init), consul-connect-inject-init (init)
2022-10-16T13:25:08.620Z [INFO]  entrypoint: entrypoint starting: deployment_id=01GFGFF53CC3WRCYEV6YAPW2P4 instance_id=01GFGFGFCCGRAH5DWE9V13MPZ8 args=["/cnb/lifecycle/launcher"]
2022-10-16T13:25:08.624Z [INFO]  entrypoint: entrypoint version: full_string=v0.10.0 version=v0.10.0 prerelease="" metadata="" revision=""
2022-10-16T13:25:08.624Z [DEBUG] entrypoint: connecting to server: addr=aad285856a77c4230afd470bbbdabe23-710024930.eu-west-1.elb.amazonaws.com:9701 tls=true tls_skip_verify=true
2022-10-16T13:25:08.625Z [DEBUG] entrypoint: waiting on server connection state to become ready
2022-10-16T13:25:08.625Z [TRACE] entrypoint: connection state: state=IDLE
2022-10-16T13:25:08.627Z [DEBUG] entrypoint.child: waiting for stateChildReady to flip to true
2022-10-16T13:25:08.627Z [TRACE] entrypoint: connection state: state=CONNECTING
2022-10-16T13:25:08.633Z [TRACE] entrypoint: connection state: state=TRANSIENT_FAILURE
2022-10-16T13:25:08.633Z [WARN]  entrypoint: failed to connect to the server, temporary network error
2022-10-16T13:25:08.633Z [WARN]  entrypoint.init: server unavailable, will retry in the background
2022-10-16T13:25:08.633Z [DEBUG] entrypoint: connecting to server: addr=aad285856a77c4230afd470bbbdabe23-710024930.eu-west-1.elb.amazonaws.com:9701 tls=true tls_skip_verify=true
2022-10-16T13:25:08.633Z [DEBUG] entrypoint: waiting on server connection state to become ready
2022-10-16T13:25:08.633Z [TRACE] entrypoint: connection state: state=IDLE
2022-10-16T13:25:08.633Z [DEBUG] entrypoint.child: starting child command watch loop
2022-10-16T13:25:08.633Z [DEBUG] entrypoint.child: child command received
2022-10-16T13:25:08.633Z [INFO]  entrypoint.child: starting child process: args=["/cnb/lifecycle/launcher"] cmd=/cnb/lifecycle/launcher
2022-10-16T13:25:08.636Z [TRACE] entrypoint: connection state: state=CONNECTING
2022-10-16T13:25:08.640Z [TRACE] entrypoint: connection state: state=TRANSIENT_FAILURE
2022-10-16T13:25:08.647Z [TRACE] entrypoint.log: sending line: line="awaiting signal"
awaiting signal
2022-10-16T13:25:09.640Z [TRACE] entrypoint: connection state: state=IDLE

logs of envoy container

> kubectl logs  hashinetes-ci-deployment-96b9445b5-dm6j7 -c envoy-sidecar
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:390] initializing epoch 0 (base id=0, hot restart version=11.104)
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:392] statically linked extensions:
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_session: envoy.http.stateful_session.cookie
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.regex_engines: envoy.regex_engines.google_re2
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.http.cache: envoy.extensions.http.cache.simple
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.common.key_value: envoy.key_value.file_based
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.request_id: envoy.request_id.uuid
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.resolvers: envoy.ip
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.protocols: dubbo
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2022-10-16 13:25:08.741][1][info][main] [source/server/server.cc:394]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.health_checkers: envoy.health_checkers.redis
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.matching.action: composite-action, skip
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   network.connection.client: default, envoy_internal
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2022-10-16 13:25:08.742][1][info][main] [source/server/server.cc:394]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2022-10-16 13:25:08.746][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:08.746][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:08.746][1][info][main] [source/server/server.cc:442] HTTP header map info:
[2022-10-16 13:25:08.748][1][info][main] [source/server/server.cc:445]   request header map: 656 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-proto,x-ot-span-context,x-request-id
[2022-10-16 13:25:08.748][1][info][main] [source/server/server.cc:445]   request trailer map: 128 bytes: 
[2022-10-16 13:25:08.748][1][info][main] [source/server/server.cc:445]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2022-10-16 13:25:08.748][1][info][main] [source/server/server.cc:445]   response trailer map: 152 bytes: grpc-message,grpc-status
[2022-10-16 13:25:08.758][1][info][main] [source/server/server.cc:811] runtime: layers:
  - name: base
    static_layer:
      re2.max_program_size.error_level: 1048576
[2022-10-16 13:25:08.759][1][info][admin] [source/server/admin/admin.cc:67] admin address: 127.0.0.1:19000
[2022-10-16 13:25:08.760][1][info][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:531] c-ares library initialized.
[2022-10-16 13:25:08.760][1][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
[2022-10-16 13:25:08.760][1][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
[2022-10-16 13:25:08.760][1][info][config] [source/server/configuration_impl.cc:97] loading 1 cluster(s)
[2022-10-16 13:25:08.843][1][info][config] [source/server/configuration_impl.cc:101] loading 0 listener(s)
[2022-10-16 13:25:08.843][1][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
[2022-10-16 13:25:08.843][1][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2022-10-16 13:25:08.843][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:221] cm init: initializing cds
[2022-10-16 13:25:08.844][1][warning][main] [source/server/server.cc:786] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2022-10-16 13:25:08.844][1][info][main] [source/server/server.cc:907] starting main dispatch loop
[2022-10-16 13:25:08.856][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 5 cluster(s), remove 0 cluster(s)
[2022-10-16 13:25:08.857][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:08.959][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:09.148][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:09.250][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 5 cluster(s), skipped 0 unmodified cluster(s)
[2022-10-16 13:25:09.250][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:195] cm init: initializing secondary clusters
[2022-10-16 13:25:09.253][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:225] cm init: all clusters initialized
[2022-10-16 13:25:09.253][1][info][main] [source/server/server.cc:888] all clusters initialized. initializing init manager
[2022-10-16 13:25:09.258][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:09.258][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-16 13:25:09.259][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener:10.0.1.161:20000'
[2022-10-16 13:25:09.276][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'outbound_listener:127.0.0.1:15001'
[2022-10-16 13:25:09.276][1][info][config] [source/server/listener_manager_impl.cc:841] all dependencies initialized. starting workers

Here are the environment variables in the application container

heroku@hashinetes-ci-deployment-96b9445b5-dm6j7:/$ env | grep -i waypoint
WAYPOINT_SERVER_TLS=1
WAYPOINT_LOG_LEVEL=TRACE
WAYPOINT_CEB_INVITE_TOKEN=z9xv4tYJPs56azWjBqUNCoBKA4jRWjfoaBMA2Fj86UMz4SPTgZFWuc7vc3Z78sgzbVTyKRfMPDKmTLxhJ7FdNK3GbGC1isjzBcUFjZDDpCQA3XU9Gbdk4QtmWK1ee8J7FGevCA4B1k2aGkLZz4GuxcQSHrzA74v9Njc9X1mwj3ZSgxearags6wT31fYqL3a1NnqcakZbWZvxsmTRq1b89GdvyKfdjQJXcXfn7v43pEbnQUvME99bdh2
WAYPOINT_SERVER_TLS_SKIP_VERIFY=1
WAYPOINT_SERVER_ADDR=aad285856a77c4230afd470bbbdabe23-710024930.eu-west-1.elb.amazonaws.com:9701
WAYPOINT_DEPLOYMENT_ID=01GFGFF53CC3WRCYEV6YAPW2P4

If launch additional waypoint entry point inside the container after connecting to it with kubectl exec I can see it connects successfully to waypoint server.

heroku@hashinetes-ci-deployment-96b9445b5-dm6j7:/$ /waypoint-entrypoint /bin/bash               
2022-10-16T13:33:07.541Z [INFO]  entrypoint: entrypoint starting: deployment_id=01GFGFF53CC3WRCYEV6YAPW2P4 instance_id=01GFGFZ32MZSEJ669C6HH92D8C args=["/bin/bash"]
2022-10-16T13:33:07.541Z [INFO]  entrypoint: entrypoint version: full_string=v0.10.0 version=v0.10.0 prerelease="" metadata="" revision=""
2022-10-16T13:33:07.541Z [DEBUG] entrypoint: connecting to server: addr=aad285856a77c4230afd470bbbdabe23-710024930.eu-west-1.elb.amazonaws.com:9701 tls=true tls_skip_verify=true
2022-10-16T13:33:07.541Z [DEBUG] entrypoint: waiting on server connection state to become ready
2022-10-16T13:33:07.541Z [TRACE] entrypoint: connection state: state=IDLE
2022-10-16T13:33:07.543Z [DEBUG] entrypoint.child: waiting for stateChildReady to flip to true
2022-10-16T13:33:07.543Z [TRACE] entrypoint: connection state: state=CONNECTING
2022-10-16T13:33:07.565Z [TRACE] entrypoint: connection state: state=READY
2022-10-16T13:33:07.566Z [DEBUG] entrypoint: connection is ready
2022-10-16T13:33:07.566Z [DEBUG] entrypoint: converting invite token to login token
2022-10-16T13:33:07.572Z [DEBUG] entrypoint: reconnecting to server with authentication
2022-10-16T13:33:07.572Z [TRACE] entrypoint: requesting version info from server
2022-10-16T13:33:07.598Z [INFO]  entrypoint: server version info: version=v0.10.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1 auth_method=token
2022-10-16T13:33:07.598Z [DEBUG] entrypoint: negotiated entrypoint protocol version: version=1
2022-10-16T13:33:07.598Z [DEBUG] entrypoint.config: registering instance, requesting config
2022-10-16T13:33:07.599Z [DEBUG] entrypoint.log: connecting to log stream
2022-10-16T13:33:07.603Z [TRACE] entrypoint.log: log stream connected
2022-10-16T13:33:07.608Z [TRACE] entrypoint: Starting a inlinekeepalive interceptor for request: method=/hashicorp.waypoint.Waypoint/EntrypointLogStream
2022-10-16T13:33:08.239Z [DEBUG] entrypoint.config_recv: first config received, switching config state to true
2022-10-16T13:33:08.239Z [DEBUG] entrypoint.config_recv: new configuration received
2022-10-16T13:33:08.239Z [TRACE] entrypoint.config.watcher.watchloop: got source config update but ignoring since there is no diff
2022-10-16T13:33:08.239Z [DEBUG] entrypoint.config.watcher.watchloop: new config variables received, scheduling refresh
2022-10-16T13:33:08.239Z [TRACE] entrypoint.config.watcher.watchloop: calculating changes between old and new config
2022-10-16T13:33:08.239Z [DEBUG] entrypoint.url: url service enabled, configuring: addr=https://control.hzn.network service_port=8080 labels=waypoint/workspace=default,waypoint.hashicorp.com/app=hashinetes-ci,waypoint.hashicorp.com/project=hashinetes-ci,waypoint.hashicorp.com/workspace=default,waypoint.hashicorp.com/instance-id=01GFGFZ32MZSEJ669C6HH92D8C,:deployment=v1,:deployment-order=01gfgff53cc3wrcyev6yapw2p4
2022-10-16T13:33:08.253Z [DEBUG] entrypoint.url: discovering hubs
2022-10-16T13:33:08.254Z [DEBUG] entrypoint.url: refreshing data
2022-10-16T13:33:08.740Z [TRACE] entrypoint.config.watcher.watchloop: refreshing app configuration
2022-10-16T13:33:08.740Z [DEBUG] entrypoint.config.watcher.watchloop: new configuration computed
2022-10-16T13:33:08.740Z [TRACE] entrypoint.config.watcher: received new config
2022-10-16T13:33:08.740Z [INFO]  entrypoint.config.watcher: env vars changed, sending new child command
2022-10-16T13:33:08.740Z [DEBUG] entrypoint.child: starting child command watch loop
2022-10-16T13:33:08.740Z [DEBUG] entrypoint.child: child command received
2022-10-16T13:33:08.740Z [INFO]  entrypoint.child: starting child process: args=["/bin/bash"] cmd=/bin/bash
2022-10-16T13:33:09.004Z [DEBUG] entrypoint.url.agent: connecting to hub: addr=52.12.2.131:443
2022-10-16T13:33:09.463Z [DEBUG] entrypoint.url.agent: connection latency: latency=215.343688ms
2022-10-16T13:33:09.463Z [DEBUG] entrypoint.url.agent: connected successfully: status=connected latency=215.343688ms skew=61.743196ms
2022-10-16T13:33:13.769Z [DEBUG] entrypoint.url.agent: connected to hub: addr=52.12.2.131:443
2022-10-16T13:33:23.741Z [TRACE] entrypoint.config.watcher.watchloop: refreshing app configuration
2022-10-16T13:33:23.741Z [TRACE] entrypoint.config.watcher.watchloop: app configuration unchanged
1 Like

@lev2 I was able to reproduce this. I’m using the kubernetes plugin instead of kubernetes-apply, but I did need to create a service beforehand with the label selector for what I knew would be the new selector of the next deployment in my cluster.

Anyway, after successful deployment of the app, with the Consul Connect sidecar deployed alongside it, the entrypoint did not connect. When I exec’d to the pod though, starting up the entrypoint was successful:

$ /waypoint-entrypoint /bin/bash
2022-10-28T18:58:05.962Z [INFO]  entrypoint: entrypoint starting: deployment_id=01GGFY2VF7W3Z8PP0V4RN1X9QN instance_id=01GGFZARGA7T2K7R62PP4P2DM9 args=["/bin/bash"]
2022-10-28T18:58:05.962Z [INFO]  entrypoint: entrypoint version: full_string=v0.10.2 version=v0.10.2 prerelease="" metadata="" revision=""
2022-10-28T18:58:05.962Z [DEBUG] entrypoint: connecting to server: addr=address:9701 tls=true tls_skip_verify=true
2022-10-28T18:58:05.962Z [DEBUG] entrypoint: waiting on server connection state to become ready
2022-10-28T18:58:05.963Z [DEBUG] entrypoint.child: waiting for stateChildReady to flip to true
2022-10-28T18:58:05.985Z [DEBUG] entrypoint: connection is ready
2022-10-28T18:58:05.985Z [DEBUG] entrypoint: converting invite token to login token
2022-10-28T18:58:06.026Z [DEBUG] entrypoint: reconnecting to server with authentication
2022-10-28T18:58:06.048Z [INFO]  entrypoint: server version info: version=v0.10.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1 auth_method=token
2022-10-28T18:58:06.048Z [DEBUG] entrypoint: negotiated entrypoint protocol version: version=1
2022-10-28T18:58:06.048Z [DEBUG] entrypoint.config: registering instance, requesting config
2022-10-28T18:58:06.048Z [DEBUG] entrypoint.log: connecting to log stream
2022-10-28T18:58:06.467Z [DEBUG] entrypoint.config_recv: first config received, switching config state to true
2022-10-28T18:58:06.467Z [DEBUG] entrypoint.config_recv: new configuration received
2022-10-28T18:58:06.467Z [DEBUG] entrypoint.config.watcher.watchloop: new config variables received, scheduling refresh
2022-10-28T18:58:06.467Z [DEBUG] entrypoint.url: url service enabled, configuring: addr=https://control.hzn.network service_port=3000 labels=waypoint/workspace=default,service=kubernetes-nodejs-web,env=dev,waypoint.hashicorp.com/app=kubernetes-nodejs-web,waypoint.hashicorp.com/project=kubernetes-nodejs,waypoint.hashicorp.com/workspace=default,waypoint.hashicorp.com/instance-id=01GGFZARGA7T2K7R62PP4P2DM9,:deployment=v7,:deployment-order=01ggfy2vf7w3z8pp0v4rn1x9qn
2022-10-28T18:58:06.469Z [DEBUG] entrypoint.url: discovering hubs
2022-10-28T18:58:06.469Z [DEBUG] entrypoint.url: refreshing data
2022-10-28T18:58:06.968Z [DEBUG] entrypoint.config.watcher.watchloop: new configuration computed
2022-10-28T18:58:06.968Z [INFO]  entrypoint.config.watcher: env vars changed, sending new child command
2022-10-28T18:58:06.968Z [DEBUG] entrypoint.child: starting child command watch loop
2022-10-28T18:58:06.968Z [DEBUG] entrypoint.child: child command received
2022-10-28T18:58:06.968Z [INFO]  entrypoint.child: starting child process: args=["/bin/bash"] cmd=/bin/bash
2022-10-28T18:58:07.073Z [DEBUG] entrypoint.url.agent: connecting to hub: addr=52.12.2.131:443
2022-10-28T18:58:07.358Z [DEBUG] entrypoint.url.agent: connection latency: latency=128.719041ms
2022-10-28T18:58:07.358Z [DEBUG] entrypoint.url.agent: connected successfully: status=connected latency=128.719041ms skew=57.539024ms
2022-10-28T18:58:11.913Z [DEBUG] entrypoint.url.agent: connected to hub: addr=52.12.2.131:443

Whilst looking deeper, I learned that if the environment variable WAYPOINT_CEB_SERVER_REQUIRED is set, THEN the Waypoint entrypoint retries connecting to the server. Otherwise, it does not appear to retry. Unfortunately in my case, enabling retries did not result in a successful connection from the CEB to the server, but for now I recommend trying that in your environment whilst I continue to debug mine. :slight_smile:

FYI, the logs from an entrypoint that IS retrying look like this:

2022-10-28T19:12:49.652Z [INFO]  entrypoint: entrypoint starting: deployment_id=01GGG05D2KTSJX6S4RP0AY1X8P instance_id=01GGG05QFMXTQ3ECNQF0JSDY4P args=["/cnb/process/web"]
2022-10-28T19:12:49.652Z [INFO]  entrypoint: entrypoint version: full_string=v0.10.2 version=v0.10.2 prerelease="" metadata="" revision=""
2022-10-28T19:12:49.652Z [DEBUG] entrypoint: connecting to server: addr=address:9701 tls=true tls_skip_verify=true
2022-10-28T19:12:49.652Z [DEBUG] entrypoint: waiting on server connection state to become ready
2022-10-28T19:12:49.653Z [DEBUG] entrypoint.child: waiting for stateChildReady to flip to true
2022-10-28T19:12:49.653Z [WARN]  entrypoint: failed to connect to the server, temporary network error
2022-10-28T19:12:49.653Z [WARN]  entrypoint.init: server unavailable but ceb configured to require it, retrying synchronously
2022-10-28T19:12:49.653Z [DEBUG] entrypoint: connecting to server: addr=address:9701 tls=true tls_skip_verify=true
2022-10-28T19:12:49.653Z [DEBUG] entrypoint: waiting on server connection state to become ready