Hi,
I’m attempting to run consul-ESM on ECS , however I’m not sure if there is any recommended compute resources requirements(Memory, CPU) to reserve for the ESM container .
Could someone clarify or point me to where it specifies in the documentation?
Thanks!
eikenb
November 3, 2022, 6:57pm
2
Hey @mohamed.ammouchi ,
I’m sorry but there is no documentation on required compute resources needed. ESM doesn’t require a ton, but what it does need would go up with the number of external services it was monitoring as well as the general size of your cluster. My best guesstimate would be to start with something on the smaller size and see how it does. If it handles it at first it should be fine… if it gets worse over time that’d be a bug.
Hope this helps.
Hey @eikenb ,
It seems that ESM is not able to fetch and update health checks on all namespaces, I tried with both version 0.6.0 and 0.6.1 and the health check is only performed against the default namespace , is this a known issue.
I tried also to run the latest version 0.6.2 but same result with additional log warning
consul-esm: Error updating check status in Consul: error="Failed request: Failed to parse body: json: unknown field \"ExposedPort\""
Thanks
eikenb
November 8, 2022, 7:15pm
4
Sorry @mohamed.ammouchi , there are a couple known issues with namespaces support in ESM (below). But… they got bumped internally so I’ll be starting work on them later this week or next. So there should be a fix out soon.
opened 06:32PM - 31 Oct 22 UTC
bug
consul
With Consul Enterprise, if there is no Consul Namespace created, Consul ESM will… crash when starting up.
#### Reproduction Steps
1. Remove all Consul namespaces
2. Run Consul-ESM
### Environment
Consul-ESM: v0.6.2
Consul: v1.13.0+ent
### Output of consul Namespaces
```
$ consul namespace list -format json
[]
```
### Actual result
```
2022-10-31T19:02:40.883+0100 [INFO] consul-esm: Trying to obtain leadership...
2022-10-31T19:02:40.892+0100 [INFO] consul-esm: Obtained leadership
2022-10-31T19:02:40.893+0100 [INFO] consul-esm: Updating external node list: items=8
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbe15e9]
goroutine 39 [running]:
main.(*Agent).getServiceInstances(0xc00010ef80, 0xc0005c9ee8)
/home/runner/work/consul-esm/consul-esm/leader.go:303 +0x289
main.(*Agent).watchServiceInstances(0xc00010ef80, 0x0?, 0xc00055c180)
/home/runner/work/consul-esm/consul-esm/leader.go:266 +0x179
created by main.(*Agent).computeWatchedNodes
/home/runner/work/consul-esm/consul-esm/leader.go:125 +0x165
```
### Workaround
Just by adding a Namespace in Consul:
```
consul namespace create -name foobar
```
opened 09:56AM - 13 Jun 22 UTC
bug
**Describe the bug**
consul-esm 0.6.0 & consul-esm 0.6.1 actually doesn't check… services from all namespaces
**To Reproduce**
Following reproduction steps what customer suggested, I tried to reproduce the steps and observe the same issue.
* Register a node with meta external-node=true (e.g. using terraform, or manually via api)
```
resource "consul_node" "test" {
name = "external-test"
address = "127.0.0.1"
meta = {
"external-node" = "true"
}
}
```
* Register a service against this node within a consul namespace (e.g. with terraform, or manually via api); nothing is actually running on localhost:1234
```
resource "consul_service" "test" {
name = "test"
node = consul_node.test
port = 1234
check {
check_id = "test:test"
name = "test"
status = "passing"
tcp = "localhost:1234"
interval = "30s"
timeout = "5s"
}
}
```
* Run consul-esm with a consul policy that gives it read privileges to all namespaces, and all services in all namespaces (e.g. using a global-management token)
* Expect the external node to be marked as critical, due to the test service not getting any response to probes on localhost:1234. Observe that the check never changes state from passing, and that logs do not show consul-esm adding a check for the test service at all.
**Expected behavior**
There should be more checks detected, as they have other services registered in non-default namespaces. They have added a service registration with an initial check result of passing but which is not actually running, so they would expect the service check to be quickly switched to critical by consul-esm, but that is not happening.
**Environment:**
- Product Version: consul-esm 0.6.0 & 0.6.1
**Server configuration file(s) and read output of any relevant mount/role/etc configuration:**
Attached in repro steps
**Is there a workaround? If so, how satisfied is the customer with the workaround?**
NA
**Additional context**
Customer has highlighted following things what they observed in last ticket [#53909](https://hashicorp.zendesk.com/agent/tickets/53909) raised by him, where same functionality of checks for all services across namespaces was discussed. However, with consul-esm 0.6.0 & 0.6.1 it was not being addressed. Besides, customer has highlighted couple of inputs by referring to codebase.
```
I think what's actually been implemented is the ability for the consul-esm leader to find instances of consul-esm service across all namespaces for the purpose of dividing the check workload across all (healthy) instances:
https://github.com/hashicorp/consul-esm/blob/6302b2f315802949f7fec72b35fe4c738eaebffd/leader.go#L279 enumerates all workspaces looking for instances of a.Config.Service, which defaults to "consul-esm" (https://github.com/hashicorp/consul-esm/blob/main/config.go#L113).
However, https://github.com/hashicorp/consul-esm/blob/6302b2f315802949f7fec72b35fe4c738eaebffd/agent.go#L459 which does the work of checking the catalog for services to be monitored has an explicit comment:
"All ESM health checks are node checks and in the 'default' namespace"
Where the API to is made to fetch a list of services without specifying a namespace, so defaults to "default" namespace. It's also not carried out inside a loop that would allow it to check more than one namespace. I think liunes ~459-470 (of git master) need to be wrapped up in a loop iterating over each namespace.
Given the changelog comment
"Add support for Consul Namespaces and controlling which namespaces ESM monitors. [GH-115]"
I think it was intended that services from all namespaces be enumerated and subsequently monitored, but it was not implemented correctly.
```
Regards,
Himanshu Sharma
Sr. Support Engineer | Consul
HashiCorp
Follow those issues or watch for the next release of consul-esm (v0.7.0 or v0.6.3).
Hope this helps.