Wordpress service can't connect with MariaDB service through Consul Connect

Consul
1

I’m having a serious problem with connecting to MariaDB through Consul Connect.

I’m using Nomad to create the services with the proxies, with the following job definition:

job "wordpress" {
  type = "service"
  datacenters = ["dc1"]

  group "server" {
    network {
      mode = "bridge"
      port "http" {
        static = 8080
        to = 80
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "wordpress"
      }
      
      env {
        WORDPRESS_DB_HOST       = "${NOMAD_UPSTREAM_ADDR_database}"
        WORDPRESS_DB_USER       = "exampleuser"
        WORDPRESS_DB_PASSWORD   = "examplepass"
        WORDPRESS_DB_NAME       = "exampledb"
      }

      resources {
        cpu     = 100
        memory  = 64
        network {
          mbits = 10
        }
      }
    }

    service {
      name = "wordpress"
      tags = ["production", "wordpress"]
      port = "http"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name  = "database"
              local_bind_port   = 3306
            }
          }
        }
      }
    }
  }

  group "database" {
    network {
      mode = "bridge"
      port "db" {
        to = 3306
      }
    }

    task "database" {
      driver = "docker"

      config {
        image = "mariadb"
      }

      env {
        MYSQL_RANDOM_ROOT_PASSWORD  = "yes"
        MYSQL_INITDB_SKIP_TZINFO    = "yes"
        MYSQL_DATABASE              = "exampledb"
      	MYSQL_USER                  = "exampleuser"
      	MYSQL_PASSWORD              = "examplepass"
      }

      resources {
        cpu     = 100
        memory  = 128
        network {
          mbits = 10
        }
      }
    }

    service {
      name = "database"
      tags = ["production", "mariadb"]
      port = "db"

      connect {
        sidecar_service {}
      }
    }
  }
}

However, it seems that the server can’t reach the database.


MySQL Connection Error: (2006) MySQL server has gone away
[25-Aug-2020 10:46:53 UTC] PHP Warning:  mysqli::__construct(): Error while reading greeting packet. PID=187 in Standard input code on line 22
[25-Aug-2020 10:46:53 UTC] PHP Warning:  mysqli::__construct(): (HY000/2006): MySQL server has gone away in Standard input code on line 22

MySQL Connection Error: (2006) MySQL server has gone away

WARNING: unable to establish a database connection to '127.0.0.1:3306'
  continuing anyways (which might have unexpected results)

And the logs of the server and database proxies shows that some sort of TLS issue is happening, but I’ve got no clue how to solve this problem.

Server Proxy Logs

[2020-08-25 12:20:35.841][18][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:344] [C1229] Creating connection to cluster database.default.dc1.internal.0198bec5-d0b4-332c-973e-372808379192.consul
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:82] creating a new connection
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:362] [C1230] connecting
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:704] [C1230] connecting to 172.29.168.233:29307
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:713] [C1230] connection in progress
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:108] queueing request due to no available connections
[2020-08-25 12:20:35.841][18][debug][main] [source/server/connection_handler_impl.cc:280] [C1229] new connection
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1229] socket event: 2
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1229] write ready
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1230] socket event: 2
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1230] write ready
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:552] [C1230] connected
[2020-08-25 12:20:35.841][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1230] handshake error: 2
[2020-08-25 12:20:35.842][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1230] socket event: 3
[2020-08-25 12:20:35.842][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1230] write ready
[2020-08-25 12:20:35.842][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1230] handshake error: 5
[2020-08-25 12:20:35.842][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C1230] 
[2020-08-25 12:20:35.842][18][debug][connection] [source/common/network/connection_impl.cc:190] [C1230] closing socket: 0
[2020-08-25 12:20:35.842][18][debug][pool] [source/common/tcp/conn_pool.cc:123] [C1230] client disconnected

Database Proxy Logs

[2020-08-25 12:26:07.093][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:201] [C927] new tcp proxy session
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:290] [C927] readDisable: enabled=true disable=true
[2020-08-25 12:26:07.093][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:344] [C927] Creating connection to cluster local_app
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:82] creating a new connection
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:362] [C928] connecting
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:704] [C928] connecting to 127.0.0.1:26344
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:713] [C928] connection in progress
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:108] queueing request due to no available connections
[2020-08-25 12:26:07.093][15][debug][main] [source/server/connection_handler_impl.cc:280] [C927] new connection
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:458] [C927] socket event: 2
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:543] [C927] write ready
[2020-08-25 12:26:07.093][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C927] handshake error: 5
[2020-08-25 12:26:07.093][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C927] 
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:190] [C927] closing socket: 0
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:204] canceling pending request
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:212] canceling pending connection
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:101] [C928] closing data_to_write=0 type=1
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:190] [C928] closing socket: 1
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:123] [C928] client disconnected
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
[2020-08-25 12:26:07.093][15][debug][main] [source/server/connection_handler_impl.cc:80] [C927] adding to cleanup list
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=2)
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:76] clearing deferred deletion list (size=2)
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:236] [C928] connection destroyed
2

I’m also having this issue.

3

Hi @luisfurnas and @brbva , with just a few tweaks to the Nomad job file I am able to get the wordpress instance talking with the database just fine. (db port configuration did not make sense) Example:

job "wordpress" {
  datacenters = ["dc1"]

  group "database" {
    network {
      mode = "bridge"
    }

    service {
      name = "database"
      port = "3306"
      tags = ["production", "mariadb"]

      connect {
        sidecar_service {}
      }
    }

    task "database" {
      driver = "docker"

      config {
        image = "mariadb"
      }

      env {
        MYSQL_RANDOM_ROOT_PASSWORD = "yes"
        MYSQL_INITDB_SKIP_TZINFO   = "yes"
        MYSQL_DATABASE             = "exampledb"
        MYSQL_USER                 = "exampleuser"
        MYSQL_PASSWORD             = "examplepass"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }

  group "server" {
    network {
      mode = "bridge"

      port "http" {
        static = 8080
        to     = 80
      }
    }

    service {
      name = "wordpress"
      port = "8080"
      tags = ["production", "wordpress"]

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "database"
              local_bind_port  = 3306
            }
          }
        }
      }
    }


    task "server" {
      driver = "docker"

      config {
        image = "wordpress"
      }

      env {
        WORDPRESS_DB_HOST     = "${NOMAD_UPSTREAM_ADDR_database}"
        WORDPRESS_DB_USER     = "exampleuser"
        WORDPRESS_DB_PASSWORD = "examplepass"
        WORDPRESS_DB_NAME     = "exampledb"
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}
4

@shoenig
The example you provided does not work with nomad 1.0.2 and consul 1.9.3, even with -dev.

Looking for a solution to this problem for hours already.

EDIT: works when using -dev-connect.