Vault as Connect CA Error intermediate expired: certificate expired

Consul
, ,
1

Hi all, we want to use Vault as Connect CA to centralized our TLS Creation using Vault.

we got this following error in our consul-client :

Aug 26 15:00:41 ip-10-110-24-154 consul[13990]: 2022-08-26T15:00:41.623Z [ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=10.110.46.190:8300 error="rpcinsecure error making call: rpcinsecure error making call: intermediate expired: certificate expired, expiration date: 2022-08-07 20:55:57 +0000 UTC "
Aug 26 15:00:41 ip-10-110-24-154 consul[13990]: 2022-08-26T15:00:41.643Z [ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=10.110.20.113:8300 error="rpcinsecure error making call: intermediate expired: certificate expired, expiration date: 2022-08-07 20:55:57 +0000 UTC "
Aug 26 15:00:41 ip-10-110-24-154 consul[13990]: 2022-08-26T15:00:41.671Z [ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=10.110.4.54:8300 error="rpcinsecure error making call: rpcinsecure error making call: intermediate expired: certificate expired, expiration date: 2022-08-07 20:55:57 +0000 UTC "
Aug 26 15:00:41 ip-10-110-24-154 consul[13990]: 2022-08-26T15:00:41.671Z [ERROR] agent.auto_config: No servers successfully responded to the auto-encrypt request

our Consul Client config be like :

client.hcl

server           = false
bind_addr        = "0.0.0.0"
client_addr      = "0.0.0.0"
advertise_addr   = "{{ GetPrivateIP }}"
encrypt          = "{{redacted}}"
log_level        = "INFO"
datacenter       = "aws-us-west-2"
data_dir         = "/opt/consul/data"

retry_join = [
  "provider=\"aws\" region=\"us-west-2\" tag_key=\"retry_join\" tag_value=\"consul-server-001\" addr_type=\"private_v4\""
]

ports {
  serf_lan = 8301
  http     = 8500
  https    = 8501
  dns      = 8600
}

tls {
  defaults {
    verify_outgoing = true
    verify_incoming = true
    ca_file         = "/opt/consul/tls/ca.crt"
  }

  internal_rpc {
    verify_server_hostname = true
  }
}

acl {
  enabled                  = true
  default_policy           = "deny"
  down_policy              = "extend-cache"
  enable_token_persistence = true
  tokens {
    default = "{{redacted}}"
    agent   = "{{redacted}}"
  }
}

connect {
  enabled     = true
}

unique.hcl

node_name      = "nomad-server-aws-ip-10-110-24-154"
node_meta {
  # {{redacted}}
  host_ip       = "10.110.24.154"
}

auto_encrypt {
  tls = true
  ip_san = [
    "10.110.24.154"
  ]
}

Consul Server config :
server.hcl

server           = true
bind_addr        = "0.0.0.0"
client_addr      = "0.0.0.0"
advertise_addr   = "{{ GetPrivateIP }}"
encrypt          = "{{redacted}}"
log_level        = "INFO"
bootstrap_expect = 3
datacenter       = "aws-us-west-2"
data_dir         = "/opt/consul/data"
domain           = "consul"

retry_join = [
  "provider=\"aws\" region=\"us-west-2\" tag_key=\"retry_join\" tag_value=\"consul-server-001\" addr_type=\"private_v4\""
]

enable_central_service_config = true
auto_reload_config            = true
leave_on_terminate            = true
rejoin_after_leave            = true
skip_leave_on_interrupt       = true

ports {
  server   = 8300
  serf_lan = 8301
  serf_wan = 8302
  http     = 8500
  https    = 8501
  dns      = 8600
}

tls {
  defaults {
    verify_outgoing = true
    ca_file         = "/opt/consul/tls/ca.crt"
    cert_file       = "/opt/consul/tls/tls.crt"
    key_file        = "/opt/consul/tls/tls.key"
  }

  internal_rpc {
    verify_incoming        = true
    verify_server_hostname = true
  }
}

auto_encrypt {
  allow_tls = true
}

acl {
  enabled                  = true
  default_policy           = "deny"
  down_policy              = "extend-cache"
  enable_token_persistence = true
  tokens {
    initial_management = "{{redacted}}"
    agent              = "{{redacted}}"
    default            = "{{redacted}}"
  }
}

autopilot {
  cleanup_dead_servers      = true
  last_contact_threshold    = "200ms"
  max_trailing_logs         = 250
  min_quorum                = 3
  server_stabilization_time = "10s"
}

connect {
  enabled     = true
  ca_provider = "vault"
  ca_config {
    address               = "https://vault-server.internal:8200"
    root_pki_path         = "connect_root"
    intermediate_pki_path = "connect_intermediate"
    token                 = "{{redacted}}"
    ca_file               = "/opt/vault/tls/ca.crt"
    cert_file             = "/opt/vault/tls/tls.crt"
    key_file              = "/opt/vault/tls/tls.key"
    tls_server_name       = "vault"
  }
}

ui_config {
  enabled = true
}

enable_debug = false

unique.hcl

node_name      = "server-aws-ip-10-110-20-113"
node_meta {
  #{{redacted}}
  host_ip       = "10.110.20.113"
}

some server error log

Aug 26 13:38:02 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:02.428Z [INFO]  connect.ca.vault: Successfully renewed token for Vault provider
Aug 26 13:38:04 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:04.691Z [INFO]  connect.ca: Correcting stored CARoot values: previous-signing-key=37:44:06:b1:84:7d:7b:f5:d7:73:ed:3a:14:bf:2e:79:7d:ae:25:cf updated-signing-key=25:cb:ef:ad:2b:ce:2d:95:36:d8:1d:5a:75:92:81:9b:71:6f:06:84
Aug 26 13:38:04 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:04.698Z [INFO]  connect.ca: initialized primary datacenter CA with provider: provider=vault
Aug 26 13:38:04 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:04.698Z [INFO]  connect.ca: Successfully initialized the Connect CA
Aug 26 13:38:04 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:04.698Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
Aug 26 13:38:04 ip-10-110-20-113 consul[3714]: 2022-08-26T13:38:04.698Z [INFO]  agent.leader: stopped routine: routine="CA initialization"

...

Aug 26 14:38:09 ip-10-110-20-113 consul[3714]: 2022-08-26T14:38:09.115Z [INFO]  connect.ca: generated new intermediate certificate for primary datacenter
Aug 26 14:38:09 ip-10-110-20-113 consul[3714]: 2022-08-26T14:38:09.121Z [INFO]  connect.ca: updated root certificates from primary datacenter

Update 1 :

  • seem I cannot upload my vault ui config screenshot