Add option to watch only changed keys

[krestenkrab]

This was discussed before in issue #410, and with @mfischer-zd here: https://groups.google.com/forum/#!topic/consul-tool/Axow65BtSuY

If we change the behaviour to be "watch only keys with new UpdateIndex", then it is possible by just filtering in the endpoint code.

Reading through the code, here

https://github.com/hashicorp/consul/blob/master/consul/kvs_endpoint.go#L151

and here

https://github.com/hashicorp/consul/blob/master/consul/kvs_endpoint.go#L191

we see that in both cases, the result is filtered by applying the ACL. What if we simply add options and code here to filter out entries or keys that have an old ModifyIndex. That way, we effectively get values that have been updated.

This should be a relatively simple code change.

A more efficient way would be to also pass this option into the state functions so that only the updated entries are returned, but that is just an optimisation.

My use case is to let a client hold an in-memory replica of a set of KVs, allowing a local scan/query through the values. This would be possible by first listing all entries with a prefix, and then subsequently doing a blocking pathprefix get with a ?index=N&updates_only option.

[slackpad]

We may want to consider adding blocking query support to /v1/txn as part of this.

[rudle]

I'm eager to see this feature land.

My team is wondering about the proper semantic for deletions - will they show up here or will we need to do a traditional style List() and compute the deltas on the client side. Another option is to present tombstone records for anything deleted since the client's specified index.

It may see redundant to mention this, but this feature has limited value to me & my team if key deletions are unsupported.

[slackpad]

My team is wondering about the proper semantic for deletions - will they show up here or will we need to do a traditional style List() and compute the deltas on the client side. Another option is to present tombstone records for anything deleted since the client's specified index.

Yes we'd definitely want to handle deletes and present them using the tombstone data. This is the part of this feature that took it from "oh this'll take 20 minutes" to "this is a little more work" :-)

[freman]

I whole heartedly throw my support (such as it is) behind this.

Using this for tracking the active microservice version, finding out that a version has changed and updating all our endpoints on a few bytes of data vs... the whole list of versions stored in all the keys :D

[krestenkrab]

If you need 'delete semantics' as described above, you can do all deletes in two steps: one 'put tombstone', follows by the real 'delete'. I imagine the programs interacting with consul would implement this "protocol". That way, observers that only see additions/changes know to remove the entry.

My use case is for a set of clients to hold a "cache" of a subset of the data in consul, kept in a fashion that lets me search it more quickly that would be possible if I had to traverse consul using the API.

[codezninja]

is there any update on this?

[durzo]

This feature is the only thing stopping us from migrating from etcd to consul. we have tens of thousands of keys under a top level prefix we want to monitor for changes. Processing each one every time a single key changes is like looking for a needle in a haystack and wasting large amounts of processing time searching for a single modification.

it baffles me that this issue isn't more important to the consul team as it is basic functionality of its competitors.

[ChrisMcKenzie]

This is something we would like to have as well and are definitely willing to contribute if necessary

[ChrisMcKenzie]

our use case is the keyprefix is quite large so would be massively inefficient to send the entire prefix down every time a single key changes.

[blake]

@ChrisMcKenzie We are working on a new streaming backend #7547 which long-term should help with this. I recommend subscribing to #6366 for updates related to supporting this with KV.

[jsosulska]

Hello all! We would like to provide an update on this ticket. As @blake has mentioned, there is a new streaming backend that we are working on. You can subscribe to #6366 as the prerequisite ticket for all streaming work. Clicking on the label theme/streaming will also show all work related to the streaming feature set. Please feel free to 👍 the first post of any topic you find relevant!

For this particular feature, the streaming feature would need to be complete. A Consul team member will post here when things are merged, and then the implementation details can be added.

[dgdelahera]

Hello, is there any update on this?