Service-mesh, envoy, upstream config. Blocks of type "config" are not expected here

Nomad
1

I am trying to configure service-mesh using docker, nomad and consul. I need to configure the retry policy of the envoy proxy server. When registering the service through consul, I solved it through proxy.upstream[*].config section of the config.

But when I try to repeat this configuration in the service description via nomad, I get an error. That this block is not supported. Maybe there are ideas how to solve this.
I attach the test configuration of the service for nomads below

job "srvexm-test" {
   datacenters = ["mesh"]
   type = "service"
   group "test" {
     network {
       mode = "bridge"
       port "im-port" {
       static = 40200
       to = 30405
       }
       port "envoy_metrics" {
       to = 9102
       }
     }

     task "im1-test" {
       driver = "docker"

        template {
        data        = "{{ key \"services/srvexm-im/config\" }}"
        destination = "local/appsettings.json"
        }

       config {
         image = "registry.mydomain/mesh:last"
      auth {
         username = "user"
         password = "user"
      }
      volumes = [
      "local/appsettings.json:/opt/work_apps/appsettings.Production.json",
        ]

       }

     }

service {
  name = "test-service"
  port = 40200
  connect {
    sidecar_service {
      port = 20000

      proxy {
                                config {
          envoy_local_cluster_json = <<EOL
           {
             "@type": "type.googleapis.com/envoy.api.v2.Cluster",
             "name": "local_app",
             "connect_timeout": "5s",
             "circuit_breakers": {
               "thresholds": [
                 {
                   "priority": "HIGH",
                   "max_requests": 1
                 }
               ]
             },
             "load_assignment": {
              "cluster_name": "local_app",
              "endpoints": [
               {
                "lb_endpoints": [
                 {
                  "endpoint": {
                   "address": {
                    "socket_address": {
                     "address": "127.0.0.1",
                     "port_value": 40200
                    }
                   }
                  }
                 }
                ]
               }
              ]
             }
           }
        EOL
        }

        upstreams {
          destination_name = "upstream"
          local_bind_port = 9001

          config {
            envoy_cluster_json = <<EOL
              {
                "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                "name": "upstream",
                "type": "EDS",
                "eds_cluster_config": {
                  "eds_config": {
                    "ads": {}
                  }
                },
                "connect_timeout": "5s",
                "outlier_detection": {
                  "consecutive_5xx": 10,
                  "consecutive_gateway_failure": 10,
                  "base_ejection_time": "30s"
                }
              }
            EOL

            envoy_listener_json = <<EOL
              {
              "@type": "type.googleapis.com/envoy.api.v2.Listener",
              "name": "upstream:127.0.0.1:9001",
              "address": {
                "socketAddress": {
                  "address": "127.0.0.1",
                  "portValue": 9001
                }
              },
              "filterChains": [
                {
                  "filters": [
                    {
                      "name": "envoy.http_connection_manager",
                      "config": {
                        "stat_prefix": "upstream",
                        "route_config": {
                          "name": "local_route",
                          "virtual_hosts": [
                            {
                              "name": "backend",
                              "domains": ["*"],
                              "routes": [
                                {
                                  "match": {
                                    "prefix": "/"
                                  },
                                  "route": {
                                    "cluster": "upstream",
                                    "timeout": "6s",
                                    "retry_policy": {
                                      "retry_on": "5xx",
                                      "num_retries": 5,
                                      "per_try_timeout": "2s"
                                    }
                                  }
                                }
                              ]
                            }
                          ]
                        },
                        "http_filters": [
                          {
                            "name": "envoy.router",
                            "config": {}
                          }
                        ]
                      }
                    }
                  ]
                }
              ]
            }
            EOL
          }
        }
      }
    }
  }
}
}
}
Summary

This text will be hidden

1 Like
2

Facing the same issue here. The documentation talks about setting upstreams[*].config via consul service configuration. The same does not work with nomad jobs.

3

Hi @sergey.gorbatov :wave:

Would it be possible to send us the full log message so we can investigate this further?

Same for you @jawahars16 if you can.

Thanks!

4

There are no logs. The error occurs when working with the console as a reaction of nomad to the command. The text of job.nomad is presented in the post above.

root@yenisey01:~# nomad job run job.nomad
Error getting job struct: Error parsing job file from job.nomad:
job.nomad:87,11-17: Unsupported block type; Blocks of type "config" are not expected here.
5

Ahh got it, thanks.

Could you trying running it like this:

service {
  name = "test-service"
  port = 40200
  connect {
    sidecar_service {
      port = 20000

      proxy {
        config = {
          envoy_local_cluster_json = <<EOL
# ...

This attribute is a map, so it must be passed with the = instead of as a block.

6

My issue is slightly different. I am able to write config block inside proxy stanza, but not inside upstreams. According to this documentation, config block is allowed inside upstreams. I am actually trying to configure outlier detection and circuit breaking in a connect service using nomad job file.

The following code throws error - Blocks of type "config" are not expected here. on command nomad job run <filename>.

Nomad version: v1.1.4

job "docs" {
  datacenters = ["dc1"]

  group "docs" {
    network {
      mode = "bridge"
    }
    service {
      name = "docs"
      port = "5678"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "demo"
              local_bind_port  = 10082
              config {
                connect_timeout_ms = 3000
              }
            }
          }
        }
      }
    }
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo"
        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
      }
    }
  }
}
7

Yes, I also tried this design option, the result is the same.
The proxy.upstream[*].config section does not work and causes an error (specified above).
Whereas the proxy.config section is available and processed correctly.

8

Oh, that’s a double face palm for me…I see the problem now. And yes, Nomad is unable to take an upstream -> config block right now.

Could you file a feature request in our repo so we can better track this request?

Thanks!

9