Consul TLS min version 1.3 Error

Hello,

I’m trying to update my service mesh (Consul - Envoy) to use TLS minimum version 1.3 on my cluster, updating from version 1.2.

Consul Version: 1.16.6
Envoy Version: 1.26.8
I confirmed that both the Consul server and Consul agent are correctly configured to use the minimum version of TLS 1.3, but the Envoy proxy that I use as a sidecar for my services is in an unhealthy status with the log:

DeltaAggregatedResources gRPC config stream to local_agent closed since 97s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
Consul Agent Configuration:

{
  "acl": {
    "enabled": true,
    "down_policy": "async-cache",
    "default_policy": "deny",
    "tokens": {
      "default": ""
    }
  },
  "enable_central_service_config": false,
  "datacenter": "",
  "encrypt": "",
  "encrypt_verify_incoming": true,
  "encrypt_verify_outgoing": true,
  "server": false,
  "log_level": "INFO",
  "advertise_addr": "",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "data_dir": "/consul/data",
  "retry_join": [
    ""
  ],
  "auto_encrypt": {
    "tls": true,
    "ip_san": [
      ""
    ]
  },
  "tls": {  
    "defaults": {
      "ca_file": "/consul/ca.pem",
      "verify_outgoing": true,
      "verify_incoming": false,
      "tls_min_version": "TLSv1_3"
    },
    "internal_rpc": {
      "verify_server_hostname": true
    }
  },
  "leave_on_terminate": true,
  "ports": {
    "https": 8501,
    "http": -1,
    "grpc": 8502,
    "grpc_tls": 8503
  },
  "domain": "consul",
  "node_meta": {
    "env": "",
    "version": ""
  }
}

Envoy Service Configuration:

{
  "service": {
    "name": "",
    "id": "",
    "token": "",
    "address": "",
    "port": 0,
    "meta": {
      "env": "",
      "version": ""
    },
    "check": {
      "deregister_critical_service_after": "30m",
      "http": "",
      "method": "GET",
      "interval": "",
      "timeout": ""
    },
    "connect": {
      "sidecar_service": {
        "port": 21000,
        "checks": [
          {
            "name": "Connect Envoy Sidecar",
            "tcp": "",
            "interval": "10s"
          },
          {
            "id": "",
            "alias_service": ""
          }
        ],
        "proxy": {
          "config": {
            "envoy_stats_bind_addr": "0.0.0.0:19001",
            "envoy_tracing_json": "{\"http\":{\"name\":\"envoy.tracers.datadog\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.DatadogConfig\",\"collector_cluster\":\"datadog_8126\",\"service_name\":\"%NAME%\"}}}",
            "envoy_extra_static_clusters_json": "{\"connect_timeout\":\"3.000s\",\"dns_lookup_family\":\"V4_ONLY\",\"lb_policy\":\"ROUND_ROBIN\",\"load_assignment\":{\"cluster_name\":\"datadog_8126\",\"endpoints\":[{\"lb_endpoints\":[{\"endpoint\":{\"address\":{\"socket_address\":{\"address\":\"%ADDRESS%\",\"port_value\":8126,\"protocol\":\"TCP\"}}}}]}]},\"name\":\"datadog_8126\",\"type\":\"STRICT_DNS\"}"
          },
          "upstreams": []
        }
      }
    }
  }
}

Can I get some help on this issue, please? Did anyone go through the same? :pray:

Additional information, I use dockerfile entrypoint script to generate the service file for my proxy envoy and consul connect envoy command to bootstrap it.

set_proxy_configuration()
{
  ## Env variables code
  ##

  base_renderers=$(jq '.service.connect.sidecar_service.proxy.upstreams = '"${CONSUL_SERVICE_UPSTREAMS}"' |
      .service.name = "'${SERVICE_NAME}'" |
      .service.id = "'${SERVICE_ID}'" |
      .service.token = "'${CONSUL_HTTP_TOKEN}'" |
      .service.address = "'${CONTAINER_IP}'" |
      .service.port = '${SERVICE_PORT}' |
      .service.meta.env = "'${DD_ENV}'" |
      .service.meta.version = "'${DD_VERSION}'" |
      .service.connect.sidecar_service.port = '${SIDECAR_PORT}' |
      .service.check.http = "'${SERVICE_HEALTH_CHECK}'" |
      .service.check.interval = "'${SERVICE_HEALTH_CHECK_INTERVAL}'" |
      .service.check.timeout = "'${SERVICE_HEALTH_CHECK_TIMEOUT}'" |
      .service.connect.sidecar_service.checks[0].tcp = "'${SIDECAR_HEALTH_CHECK}'" |
      .service.connect.sidecar_service.checks[1].id = "'${SERVICE_ID}'-alias" |
      .service.connect.sidecar_service.checks[1].alias_service = "'${SERVICE_ID}'" |
      .service.connect.sidecar_service.proxy.config.envoy_tracing_json |=gsub("%NAME%";"'$DD_SERVICE'") |
      .service.connect.sidecar_service.proxy.config.envoy_extra_static_clusters_json |= gsub("%ADDRESS%"; "'$EC2_HOST_ADDRESS'") |
      .service.connect.sidecar_service.proxy.config.common_tls_context.tls_params.tls_minimum_protocol_version = "TLSv1_3"' ./service_config.json)

echo "Base Renderers configuration: $base_renderers"

  # Wait until Consul can be contacted
  until curl -s -k ${CONSUL_HTTP_ADDR}/v1/status/leader | grep ***; do
    echo "Waiting for Consul to start at ${CONSUL_HTTP_ADDR}."
    sleep 1
  done

  echo "Registering service with consul ${SERVICE_CONFIG_FILE}."
  consul services register ${SERVICE_CONFIG_FILE}

  consul connect envoy -sidecar-for=${SERVICE_ID} -grpc-ca-file=${CONSUL_CACERT} $ENVOY_DEBUG &
}

Kind Regards,
Joel Vaz

I also created a topic on the Envoy github and was recommended to check here for guidance with the consul connect envoy command: Error when using TLS min version 1.3 with Envoy Proxy · Issue #36181 · envoyproxy/envoy · GitHub

Hi @joel-vaz,

Welcome to the HashiCorp Forums!

Here is a workaround that you can use to get this working. The following steps are what you should be taking:

  1. Use consul connect envoy command to generate the Envoy bootstrap config
  2. Patch the bootstrap config to have the TLSv1_3 related config
  3. Launch Envoy directly using the above bootstrap config.

Eg:

# Generate bootstrap config
consul connect envoy -sidecar-for=${SERVICE_ID} -grpc-ca-file=${CONSUL_CACERT} -bootstrap > bootstrap.json

# Patch the config using jq
jq '.static_resources.clusters[0].transport_socket.typed_config.common_tls_context.tls_params = { "tls_maximum_protocol_version": "TLSv1_3" }' bootstrap.json  > tmp.json && mv tmp.json bootstrap.json

# Launch envoy
envoy --config-path bootstrap.json

This should do the trick.

Hello @Ranjandas

I applied your suggestion:

Bootstrap config file:

{
 "admin": {
   "access_log": [
     {
       "name": "envoy.access_loggers.file",
       "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
         "path": "/dev/null"
       }
     }
   ],
   "address": {
     "socket_address": {
       "address": "<>",
       "port_value": <>
     }
   }
 },
 "node": {
   "cluster": "mesh-sb-test-appliB",
   "id": "<>",
   "metadata": {
     "namespace": "default",
     "partition": "default"
   }
 },
 "layered_runtime": {
   "layers": [
     {
       "name": "base",
       "static_layer": {
         "re2.max_program_size.error_level": 1048576
       }
     }
   ]
 },
 "static_resources": {
   "clusters": [
     {
       "name": "local_agent",
       "ignore_health_on_host_removal": false,
       "connect_timeout": "1s",
       "type": "STATIC",
       "transport_socket": {
         "name": "tls",
         "typed_config": {
           "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
           "common_tls_context": {
             "validation_context": {
               "trusted_ca": {
                 "inline_string": "-----BEGIN CERTIFICATE-----\n<>\n-----END CERTIFICATE-----\n"
               }
             },
             "tls_params": {
               "tls_minimum_protocol_version": "TLSv1_3"
             }
           }
         }
       },
       "typed_extension_protocol_options": {
         "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
           "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
           "explicit_http_config": {
             "http2_protocol_options": {}
           }
         }
       },
       "loadAssignment": {
         "clusterName": "local_agent",
         "endpoints": [
           {
             "lbEndpoints": [
               {
                 "endpoint": {
                   "address": {
                     "socket_address": {
                       "address": "<>",
                       "port_value": <>
                     }
                   }
                 }
               }
             ]
           }
         ]
       }
     },
     {
       "connect_timeout": "3.000s",
       "dns_lookup_family": "V4_ONLY",
       "lb_policy": "ROUND_ROBIN",
       "load_assignment": {
         "cluster_name": "datadog_8126",
         "endpoints": [
           {
             "lb_endpoints": [
               {
                 "endpoint": {
                   "address": {
                     "socket_address": {
                       "address": "<>",
                       "port_value": <>,
                       "protocol": "TCP"
                     }
                   }
                 }
               }
             ]
           }
         ]
       },
       "name": "datadog_8126",
       "type": "STRICT_DNS"
     },
     {
       "name": "self_admin",
       "ignore_health_on_host_removal": false,
       "connect_timeout": "5s",
       "type": "STATIC",
       "typed_extension_protocol_options": {
         "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
           "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
           "explicit_http_config": {
             "http_protocol_options": {}
           }
         }
       },
       "loadAssignment": {
         "clusterName": "self_admin",
         "endpoints": [
           {
             "lbEndpoints": [
               {
                 "endpoint": {
                   "address": {
                     "socket_address": {
                       "address": "<>",
                       "port_value": <>
                     }
                   }
                 }
               }
             ]
           }
         ]
       }
     }
   ],
   "listeners": [
     {
       "name": "envoy_metrics_listener",
       "address": {
         "socket_address": {
           "address": "<>",
           "port_value": <>
         }
       },
       "filter_chains": [
         {
           "filters": [
             {
               "name": "envoy.filters.network.http_connection_manager",
               "typedConfig": {
                 "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                 "stat_prefix": "envoy_metrics",
                 "codec_type": "HTTP1",
                 "route_config": {
                   "name": "self_admin_route",
                   "virtual_hosts": [
                     {
                       "name": "self_admin",
                       "domains": [
                         "*"
                       ],
                       "routes": [
                         {
                           "match": {
                             "prefix": "/stats"
                           },
                           "route": {
                             "cluster": "self_admin",
                             "prefix_rewrite": "/stats"
                           }
                         },
                         {
                           "match": {
                             "prefix": "/"
                           },
                           "direct_response": {
                             "status": 404
                           }
                         }
                       ]
                     }
                   ]
                 },
                 "http_filters": [
                   {
                     "name": "envoy.filters.http.router",
                     "typedConfig": {
                       "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                     }
                   }
                 ]
               }
             }
           ]
         }
       ]
     }
   ]
 },
 "stats_config": {
   "stats_tags": [
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.custom_hash"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.service_subset"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.service"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.namespace"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:([^.]+)\\.)?[^.]+\\.internal[^.]*\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.partition"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.datacenter"
     },
     {
       "regex": "^cluster\\.([^.]+\\.(?:[^.]+\\.)?([^.]+)\\.external\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.peer"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.routing_type"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
       "tag_name": "consul.destination.trust_domain"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.destination.target"
     },
     {
       "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
       "tag_name": "consul.destination.full_target"
     },
     {
       "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.(([^.]+)(?:\\.[^.]+)?(?:\\.[^.]+)?\\.[^.]+\\.)",
       "tag_name": "consul.upstream.service"
     },
     {
       "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.[^.]+)?\\.([^.]+)\\.)",
       "tag_name": "consul.upstream.datacenter"
     },
     {
       "regex": "^(?:tcp|http)\\.upstream_peered\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)",
       "tag_name": "consul.upstream.peer"
     },
     {
       "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.([^.]+(?:\\.([^.]+))?(?:\\.[^.]+)?\\.[^.]+\\.)",
       "tag_name": "consul.upstream.namespace"
     },
     {
       "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.([^.]+))?\\.[^.]+\\.)",
       "tag_name": "consul.upstream.partition"
     },
     {
       "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.custom_hash"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.service_subset"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.service"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.namespace"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.datacenter"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.routing_type"
     },
     {
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
       "tag_name": "consul.trust_domain"
     },
     {
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
       "tag_name": "consul.target"
     },
     {
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
       "tag_name": "consul.full_target"
     },
     {
       "tag_name": "local_cluster",
       "fixed_value": "<>"
     },
     {
       "tag_name": "consul.source.service",
       "fixed_value": "<>"
     },
     {
       "tag_name": "consul.source.namespace",
       "fixed_value": "default"
     },
     {
       "tag_name": "consul.source.partition",
       "fixed_value": "default"
     },
     {
       "tag_name": "consul.source.datacenter",
       "fixed_value": "<>"
     }
   ],
   "use_all_default_tags": true
 },
 "tracing": {
   "http": {
     "name": "envoy.tracers.datadog",
     "typedConfig": {
       "@type": "type.googleapis.com/envoy.config.trace.v3.DatadogConfig",
       "collector_cluster": "datadog_8126",
       "service_name": "<>"
     }
   }
 },
 "dynamic_resources": {
   "lds_config": {
     "ads": {},
     "initial_fetch_timeout": "0s",
     "resource_api_version": "V3"
   },
   "cds_config": {
     "ads": {},
     "initial_fetch_timeout": "0s",
     "resource_api_version": "V3"
   },
   "ads_config": {
     "api_type": "DELTA_GRPC",
     "transport_api_version": "V3",
     "grpc_services": {
       "initial_metadata": [
         {
           "key": "x-consul-token",
           "value": "<>"
         }
       ],
       "envoy_grpc": {
         "cluster_name": "local_agent"
       }
     }
   }
 }
}

But when I attempt to start Envoy with this configuration now got the error:

[./source/common/config/grpc_stream.h:191] DeltaAggregatedResources gRPC config stream to local_agent closed since 52s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435736:SSL routines:OPENSSL_internal:NO_SUPPORTED_VERSIONS_ENABLED

Any thing I’m missing?

Thank you so much for your support here :pray:

I found a workaround for it to set both the tls min and max versions on the bootstrap configuration file ^

It didn’t work because you seemed to have used the wrong configuration compared to what I shared.

tls_minimum_protocol_version": "TLSv1_3" (minimum vs maximum).

I am glad that you got it working later. :+1:

By the way, I have raised a PR for this feature to be available natively in Consul. Envoy Bootstrap Config Support TLSv1_3 by Ranjandas · Pull Request #21777 · hashicorp/consul · GitHub