KV Get querying with index and wait returns older entries than index

Consul
1

Carried over from: https://github.com/hashicorp/consul/issues/7958 by dariopb

I couldn’t find this behavior documented but I would expect that when I query KV and I set an index I would get only new entries or not found if timeout, instead every entry is returned:

c:\projects\k3s-win\cmd\agent>curl -v -X GET  "http://127.0.0.1:8500/v1/kv/server?**recurse&index=9999&wait=2s**
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8500 (#0)
> GET /v1/kv/server?recurse&index=0&wait=2s HTTP/1.1
> Host: 127.0.0.1:8500
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
**< X-Consul-Index: 6997**
< X-Consul-Knownleader: true
< X-Consul-Lastcontact: 0
< Date: Wed, 27 May 2020 02:38:14 GMT
< Content-Length: 732
<
[{"LockIndex":0,"Key":"server/0001","Flags":0,"Value":"J3tuYW1lOjAwMDF9Jw==","CreateIndex":6188,"ModifyIndex":6188},{"LockIndex":0,"Key":"server/0001/realm-0001","Flags":0,"Value":"J3tuYW1lOjAwMDF9Jw==","CreateIndex":6990,"ModifyIndex":6990},{"LockIndex":0,"Key":"server/0001/status","Flags":0,"Value":"J3tuYW1lOjAwMDF9Jw==","CreateIndex":6997,"ModifyIndex":6997},{"LockIndex":0,"Key":"server/0001/status/realm-0001","Flags":0,"Value":"J3tuYW1lOjAwMDF9Jw==","CreateIndex":6994,"ModifyIndex":6994},{"LockIndex":0,"Key":"server/0002","Flags":0,"Value":"J3tuYW1lOjAwMDJ9Jw==","CreateIndex":6189,"ModifyIndex":6189},{"LockIndex":0,"Key":"server/0002/free","Flags":0,"Value":"J3tuYW1lOjAwMDJ9Jw==","CreateIndex":6987,"ModifyIndex":6987}]* Connection #0 to host 127.0.0.1 left intact
2

Hi Dario,

Thanks for posting. I went through our blocking query documentation which goes through how blocking queries function, and here’s what I did.

  1. Set up a Consul server in dev mode
  2. Created some keys and values {server,server/001,server/realm-001}
  3. I used your curl command
    curl -v -X GET "http://127.0.0.1:8500/v1/kv/server?recurse&index=9999&wait=2s

In the blocking query document, it specifies:

When this is provided, the HTTP request will “hang” until a change in the system occurs, or the maximum timeout is reached. A critical note is that the return of a blocking request is no guarantee of a change. It is possible that the timeout was reached or that there was an idempotent write that does not affect the result of the query.

Because the wait is achieved, and no index is met, the remainder of the query is executed. That is why you are getting all of the values, rather than an error.

I hope this helps, and happy coding!

3

Thank you jsosulska!

I did the same and came to the same conclusion: this is how it is implemented. Now, is this the way people expect it to work? If I pass an index, IMHO, it is because I expect newer items in the response (and wait if there are none and wait was supplied), plus in the current implementation it is always returning every item that matches. I did the change to support that behavior for my needs (at the kvs endpoint for now, trying to move it to the rcp layer) but wondering what is needed to change the behavior in general, if not a bug (that was closed), where is the place to open a discussion about this?

4

Hi @dariopb,

Thanks for the quick reply! I’ll address each thing as they came up.

If I pass an index, IMHO, it is because I expect newer items in the response (and wait if there are none and wait was supplied), plus in the current implementation it is always returning every item that matches.

curl -v -X GET "http://127.0.0.1:8500/v1/kv/server?**recurse**

Just to clarify, the every item that matches comes from the “recurse” being included in the query. When querying without it, it will provide the singular result that matches.

Can you provide me some more context for your use case here? I believe Consul watches, which trigger an event on the change of a resource, use blocking queries in a more managed way, and don’t rely on you needing to evaluate all of the indexs.

I did the change to support that behavior for my needs (at the kvs endpoint for now, trying to move it to the rcp layer)

Oooh, I’d love to see some steps on how you solved this. That, along with usecases, helps us figure out how more people are using our tools.

… what is needed to change the behavior in general, if not a bug (that was closed), where is the place to open a discussion about this?

You’re in the right place for discussion! Here, community members can add upvotes by commenting on the thread, and adding a <3 to the first post in the topic, and we’ll take that feedback to the team. As I mentioned earlier, a break down of your use case, and work arounds would be helpful to pass along to the core team.

As a brief aside - Consul is positioned as a service networking solution, that has KV functionality. Consul is not intended to be used as a fully featured datastore, so a more advanced data store may better fit your needs.

I look forward to your responses, and thanks for bringing this up!

5

Yes, the recurse option is what I intended to use in that case (but even not passing that you get the current entry even if there were no changes).

The use case basically is: I want to get “notified” (getting the list of new entries) when something changes in an object (or a set of objects, with recurse) or not-found/timeout if nothing changed (like the documentation states). So, for example, in that sample I have “servers”, the clients are watching that, whenever something changes to any server entry, or a new server is added, I get the response: not all of them of course, but the one(s) that changed!.

The change is a naive one to filter the results (need to move it to the rpc layer so they are not transmitted) in kvs_endpoint.go:

	// Make the RPC
	var out structs.IndexedDirEntries
	if err := s.agent.RPC(method, &args, &out); err != nil {
		return nil, err
	}
	setMeta(resp, &out.QueryMeta)

	// Only return the entries that are actually after the requested index
	if args.QueryOptions.MinQueryIndex != 0 {
		var e *structs.DirEntry
		var entries []*structs.DirEntry
		for _, e = range out.Entries {
			if e.ModifyIndex > args.QueryOptions.MinQueryIndex {
				entries = append(entries, e)
			}
		}

		out.Entries = entries
	}

Are the “watches” filtering the set then (i didn’t look at the code for those)? since if they are not, they’ll expose the same behavior (they are calling the same rest api). Watches are not very useful for me since I would prefer my clients to connect to somewhere as opposed to opening incoming ports.

I’m not trying to use consul as a generic datastore but just to hold fib entries for a networking solution, replication and “notification when something changed” are the main features I’m after.

1 Like