Without traffic signals and signage, life would not be pleasant on local streets; motorists would drive in opposing lanes and chaos would ensue. A stop sign is a reliable standard; it means the same thing no matter where it may be located.

The OpenTelemetry project is the computing industry’s attempt to provide a de facto standard for observability within cloud native applications and avoid a lot of wasted time and effort blundering around digital roads. It is a merging of the OpenTracing and OpenCensus projects so that telemetry data can be portable and useful regardless of the vendor. By creating a consolidated system with metrics, tracing and logging, users of observability systems can more easily identify failures, track running processes and analyze application-specific messages.

Over 80 companies are involved in this open-source initiative. The list includes Splunk Inc., Microsoft and Google, to name a few. After Kubernetes, OpenTelemetry has become the most popular project in the Cloud Native Computing Foundation.

“If you look at the major players in OpenTelemetry, they have a wide variety of vendor experience,” said Constance Caramanolis, software engineer at Splunk and an active contributor to the project, during an interview with SiliconANGLE. “For service owners or businesses trying to collect data and have visibility into applications, this is a really great way to provide one common framework to gather and generate all of that data.”

Easier to find failures

OpenTelemetry’s popularity within CNCF is a reflection of widespread interest in a single set of agent software that can generate data for use across multiple platforms. It would replace a current situation in which enterprises must deploy and manage agent software for separate IT management tools.

“What OpenTelemetry gives us is the first wholly acceptable industry-neutral format for tracing,” said Gregg Siegfried, research director on the Cloud and IT Operations team at Gartner Inc., in an exclusive interview with SiliconANGLE. “You’ll no longer be bound to a particular vendor’s agent.”

OpenTelemetry includes several moving parts, including API instrument code that generate traces; SDKs that collect data for processing and export; process exporters that translate the data into custom formats; and collectors that handle data filtering, batching and communication. What this means for observability is that users of the tool will be able to effectively identify failures or performance issues.

Tracing has been a complicated process because of its dependence on a journey through multiple layers of applications, paired with vague metrics and log data. OpenTelemetry addresses this complexity by unifying tracing, metrics and log data, thereby saving time.

“We’re convinced OpenTelemetry will become an industry standard for observability, benefiting from unprecedented growth over the next few years,” said Tim Tully, chief technical officer at Splunk, in a post last year. “In fact, we predict that by the end of 2022, OpenTelemetry will be the most commonly used data instrumentation and collection technology in the world.”

Taking observability out for a spin

If Tully’s prediction of common adoption by the end of next year comes true, it will require a concerted effort on the OpenTelemetry development front among major industry players. Following a beta release in March, the tracing specification moved into release candidate status in October, with a gradual rollout of API, SDK and collector functions. The current focus is on prioritizing metrics-related work and additional testing before OpenTelemetry can move into general availability.

Today, Splunk’s OpenTelemetry distribution for the OTel Collector, including client libraries for popular programming languages such as Java and Python, are readily available for use on Github. Customers can now test drive the use of the OTel Collector, which can also be deployed as a standalone service (also known as a gateway), to receive, process and export trace, metric and log data.

Meanwhile, some industry players are taking the observability framework out for a spin. Shopify Inc., one of the project’s contributors, has been moving toward putting OpenTelemetry into production for Kubernetes monitoring on Google Cloud Platform.

This particular application of the project for Kubernetes is part of an overall interest in finding ways to operationalize the complicated container orchestration technology.

“Kubernetes is a phenomenally complex beast to tame,” said Dinesh Dutt, principal and co-founder of Stardust Systems Inc. and a former Fellow at Cisco Systems Inc., in an exclusive interview with SiliconANGLE. “Making the operational efficiency of Kubernetes high is driving the popularity of OpenTelemetry.”

Amazon Web Services Inc. has also launched a preview of its AWS Distro for OpenTelemetry, designed to provide metadata collection and instrumentation for applications running a variety of AWS services.

“Teams using AWS along AppDynamics, Grafana, New Relic, Splunk or Datadog tools should be able to easily add the new insights into their regular setups, thanks to specially added exporters and other helpers,” noted one DevClass report on the October developments. “A limiting factor is the language used for application development, though, since AWS Distro for OpenTelemetry currently only works with Java apps. JavaScript and .NET tooling is soon to be added, followed by Python, Ruby, Go, C++, Erlang and Rust support.”

Interest in the OpenTelemetry project and observability frameworks is also reshaping corporate positioning in the enterprise world. Observability innovator Omnition Inc. was acquired by Splunk in 2019, and the company followed that up by purchasing application performance monitoring startups Plumbr and Rigor in October. New Relic Inc. acquired Pixie Labs Inc., a next-generation solution for tracking Kubernetes workloads, late last year.

Behind all of this activity lies the emergence of several key themes being driven by OpenTelemetry and the observability space in general. One is that this has become a fertile area for open-source tools as companies are spending considerable time and energy contributing to and running projects.

Another is that the toolbox is wide open. A CNCF study in 2020 found that half of the companies surveyed were using five or more observability tools and a third of them were involved with more than 10. Consolidation can be challenging across best-of-breed technologies, but progress has been made to manage logs and report on fine-grained information.

“This is the moment of observability,” said Caramanolis during virtual KubeCon 2020 in November. “As we build more complex systems, we need to better understand our systems, and so it is time for a project that merges multiple telemetry formats: metrics, traces and eventually logs.”

Make sure to check out theCUBE’s coverage of a special four-part CUBE Conversation series on the importance of observability and how Splunk’s history of big data analysis is helping the company address today’s monitoring challenges.

Image: BsWei