Hello,
I am trying to upgrade nomad from version 1.6.2 to 1.9.4, during the process of the upgrade as soon as I upgrade my nomad server all the microservices which are in the cluster are getting restarted. This is not actually expected as I have a nomad cluster. Do anyone faced this issue while upgrading to nomad latest version?
Hi,
I have followed the process by upgrading nomad client first and then nomad server but still facing the same issue as microservices are getting restarting. I have checked by upgrading with the following versions (1.9.0, 1.8.0, 1.7.0)
my 0.02:
1.6.x to 1.9.x is quite the jump, so this could be expected. I doubt it, but not sure though.
Q: how many servers do you have? you say server , do you have a single server?
Q: what type of driver is used for the services? raw_exec? docker?
in case of multiple servers, you should update non-leaders first (in a serial manner), let them settle for a few minutes after each node upgrade and then move to the next server.
HTH.
I’ll let someone from HashiCorp confirm if a 3 version jump should expect services restart, but I doubt it.
Hi
I have a single server.
docker driver
I am following the canary process. I have a single server with 1.6.2 I am adding 2 more servers to that cluster with 1.6.2 and then I am adding 3 more servers with 1.9.4v and then I am switching based on the canary(0) and making it as leader. But as soon as I am adding new version servers even before switching microservices are getting restarted.
The services would definitely restart if there is only one server which restarts.
NOTE: this is subject to the job heartbeat timeouts, though.
What do mean “canary” for the servers? I only knew of canary for the jobs (i.e. micro-services).
Can you cite the link you are using as reference for canary server?
There seem to be too many unknowns in your problem description.
When you say adding servers with a newer version, what is the original bootstrap_expect value? I assume 1 as you are originally running with a single server.
The upgrade process I use, I upgrade servers BEFORE clients.
When upgrading multiple servers, I upgrade the leader-server LAST.
Can you cite which docs you are using for upgrade guidelines?
HTH.
I’ll let some from HashiCorp comment if there is something else they notice from the problem description.