Consul service discovery with springboot

Consul
1

HI,

I am trying to use Consul service discovery through spring cloud dependency with group org.springframework.cloud and artifact id spring-cloud-starter-consul-discovery

It works fine most of the time, but sometimes Consul removes the live services.
After restarting the removed live service, Consul will register it once again.

Any reason for this?

Thanks,
HarshaYerasi.

2

Hi @harshayerasi,

Thanks for posting, and welcome to the forums :slight_smile: I’d like to get some more information to find out about what’s happening.

Can you please provide the service definition for a service that has been removed? Is it the same service every time, or does this happen across most/all services?

Are you able to replicate the issue consistently? Can you please post the steps for replication?

Consul should not go through and remove live services unless there is an explicit consul service deregister command, or leave or force-leave performed on the nodes where the services are hosts.

Looking forward to hearing back from you!

Jono

3

Hi @jsosulska,

Thank you for the reply,

  1. The services are spring boot apps installed as windows services.
  2. It happens with different services.
  3. The problem is not noticed all the time it is noticed occassionally as far as I have noticed.
  4. We have configured Gitlab runner to the test environment and each time the new code is committed the services will restart through restart command. After restart the services are not registered back to consul. If I login to server and manually restart the service then the service is again registered.

Hope this information is good enough.
Please let me know your thoughts on this.

4

Hi @harshayerasi,

Thanks for being patient with my questions. This is a little harder to troubleshoot as the “spring-cloud-consul” integration is maintained by Spring, and not HashiCorp directly. You may find more answers by posting on their github issues. Feel free to post the issue when it’s made, and link back to this post as well.

In the meantime, I would like to try and get some more information, to see what could possibly be causing this.

  1. Can you please post the application.yml, with the sensitive bits removed? This could be a misconfiguration between the service definition itself.
  2. What version of Consul is running on the local agent?
  3. What is the version of the plugin you are using?
  4. Can you post your Consul Agent configuration as well?
  5. Is your cluster using any sort of service mesh/proxies in front of your services?
  6. During the services disappearing, what do your Consul logs say? Please upload snippets to gist.github.com, with the sensitive bits removed.
  7. During the services disappearing, what is your resource consumption (mem and CPU)? Is there high load?
  8. How long are the apps running for when you are restarting them? Are they long running processes, or does this happen relatively quickly after restart?