We have a few services and one of them has some proxy upstreams, everything worked fine for 2 days when then envoy seems to have recreated the public listener but not the upstream ones, this is the log:
[2022-12-19 21:25:47.066][16][info][admin] [source/server/admin/admin.cc:134] admin address: 127.0.0.1:19000
[2022-12-19 21:25:47.067][16][info][config] [source/server/configuration_impl.cc:127] loading tracing configuration
[2022-12-19 21:25:47.067][16][info][config] [source/server/configuration_impl.cc:87] loading 0 static secret(s)
[2022-12-19 21:25:47.067][16][info][config] [source/server/configuration_impl.cc:93] loading 1 cluster(s)
[2022-12-19 21:25:47.119][16][info][config] [source/server/configuration_impl.cc:97] loading 0 listener(s)
[2022-12-19 21:25:47.120][16][info][config] [source/server/configuration_impl.cc:109] loading stats configuration
[2022-12-19 21:25:47.120][16][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2022-12-19 21:25:47.120][16][info][upstream] [source/common/upstream/cluster_manager_impl.cc:205] cm init: initializing cds
[2022-12-19 21:25:47.120][16][warning][main] [source/server/server.cc:761] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2022-12-19 21:25:47.121][16][info][main] [source/server/server.cc:882] starting main dispatch loop
[2022-12-19 21:25:47.139][16][info][upstream] [source/common/upstream/cds_api_helper.cc:30] cds: add 4 cluster(s), remove 0 cluster(s)
[2022-12-19 21:25:47.174][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-19 21:25:47.326][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-19 21:25:47.470][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-19 21:25:47.596][16][info][upstream] [source/common/upstream/cds_api_helper.cc:67] cds: added/updated 4 cluster(s), skipped 0 unmodified cluster(s)
[2022-12-19 21:25:47.596][16][info][upstream] [source/common/upstream/cluster_manager_impl.cc:184] cm init: initializing secondary clusters
[2022-12-19 21:25:47.599][16][info][upstream] [source/common/upstream/cluster_manager_impl.cc:209] cm init: all clusters initialized
[2022-12-19 21:25:47.599][16][info][main] [source/server/server.cc:863] all clusters initialized. initializing init manager
[2022-12-19 21:25:47.607][16][info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:20000'
[2022-12-19 21:25:47.608][16][info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'ext-service-i-xx:127.0.0.1:8000'
[2022-12-19 21:25:47.609][16][info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'ext-service-i-yy:127.0.0.1:8001'
[2022-12-19 21:25:47.610][16][info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'ext-service-i-zz?peer=ext-cluster:127.0.0.1:8002'
[2022-12-19 21:25:47.610][16][info][config] [source/server/listener_manager_impl.cc:789] all dependencies initialized. starting workers
[2022-12-19 21:40:47.600][16][info][main] [source/server/drain_manager_impl.cc:171] shutting down parent after drain
[2022-12-21 19:30:16.199][16][info][upstream] [source/common/upstream/cds_api_helper.cc:30] cds: add 3 cluster(s), remove 0 cluster(s)
[2022-12-21 19:30:16.225][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-21 19:30:16.423][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-21 19:30:16.566][16][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-12-21 19:30:16.732][16][info][upstream] [source/common/upstream/cds_api_helper.cc:67] cds: added/updated 3 cluster(s), skipped 0 unmodified cluster(s)
[2022-12-21 19:30:16.777][16][info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:20001'
you can see that after the first initial configuration it didn’t log anything until today where it only restarted the public listener and not the upstreams.
After a restart it re-registered everything and it started working again.
What can be the issue here?