streaming: Expand Streaming support to all blocking queries

[banks]

This is a placeholder issue that will be modified over time.

We are currently working on a new internal architecture to power blocking queries that is significantly more efficient. One benchmark based on an extreme but real customer workload showed the following levels of improvement for servers:

Steady CPU: 60% -> <1%, a 60x+ reduction in CPU
Memory: 420MB -> 20MB (minus 80MB baseline), a 21x reduction in Memory
Outbound Bandwidth: 2.2Gbps -> 0.5Mbps (mins 500k baseline), a 4,400x reduction in network throughput

The public API for now remains the same and this is a purely internal (between client and server agents) change.

The new architecture has been intentionally designed to roll out incrementally to make it more practical and we are closing in on the first point at which we can release a major part of Consul being powered by this.

The first use-case will be the biggest impact one for many workloads - service health watching. Once that is rolled out, this issue is a placeholder for the iterative process of rolling out the same architecture to cover eventually all other blocking APIs.

The proposed order to roll them out in terms of impact is:

• Anything replicated between DCs over WAN:
  • ACL Tokens and Policies
  • ConfigEntries
  • Intentions
• Catalog (not just Health)
• KV

Until we have completed and are ready to ship the first use-case this is a placeholder, but once we've done that and shown the patter is successful, the plan is to update this issue with a "guide" walking through the steps needed to upgrade a given API endpoint from the above list to use streaming. From there we can make individual tasks and hopefully burn them down with help from the community until all of Consul's blocking mechanisms are ported to the new streaming mechanism.

More info to come here once we are ready for that.

CC @Aestek @pierresouchay FYI.

[pierresouchay]

Great news, tell us if we can help

[banks]

tell us if we can help

Basically watch this issue and once we have work to do it will be here! Thanks!

[ChrisMcKenzie]

Any update on this feature, would love to contribute if help is needed,

[helight]

we look for this feature for a long time. we want to use this feature ASAP. thanks guys for your great work.

[jkirschner-hashicorp]

Hi all - a quick update on our progress here.

We've made significant progress on implementing streaming for the service health watching use case (see Consul 1.9 release blog post). We've also created some high-level developer-focused documents which can help guide the addition of more streaming topics in the future.

There's still work to be done to polish the service health watching use case and to improve the developer docs for streaming.

At this time, we are not yet in a state where we are ready to support or accept community contributions to implement streaming for other endpoints.

[dnlopes]

Any news on this topic? It has been over an year since the last update.

I'm particurarly interested in the KV endpoint change.

[seanmacisaac]

Another ping on this issue. We could use streaming for the KV endpoints and at some point of scale will likely need to move away from consul if it isn't supported.

[dnlopes]

Another ping on this issue. We could use streaming for the KV endpoints and at some point of scale will likely need to move away from consul if it isn't supported.

Yes, we have used consul-replicate in the past but it does not scale for our use case. A streaming solution that only outputs the deltas would be way more efficient.

[seanmacisaac]

Any progress here? Are you looking for open source contributions?