OpenTelemetry introduction

Introduction

In today’s software landscape, the prevalence of microservices architecture underscores the critical need for robust observability to ensure the seamless development and operation of projects. Fortunately, the open-source community has introduced OpenTelemetry, an Observability framework and toolkit designed to create and manage telemetry data, encompassing elements such as traces, metrics, and logs. The OpenTelemetry project develops specifications and standards for telemetry data protocols. Notably, OpenTelemetry is the successful fusion of two preceding projects, OpenTracing and OpenCensus.

OpenTelemetry:

• collectors and processors of telemetry data
• ecosystem of programming language libraries and frameworks for application instrumentation
• various additional tools, such as the OpenTelemetry Operator for Kubernetes

It is important to clarify that OpenTelemetry does not function as a comprehensive “full-stack” observability tool responsible for the storage and visualization of telemetry data.

Below, we present a comprehensive schema showing components specific to any observability system:

 ┌───────────────┐    ┌───────────────┐    ┌───────────────┐    ┌───────────────┐
 │ Application   │    │               │    │               │    │               │
 │ instrumented  │    │               │    │ Observability │    │               │
 │ to generate   ├───►│   Collector   ├───►│    data       ├───►│      UI       │
 │ observability │    │               │    │   storage     │    │               │
 │ data          │    │               │    │               │    │               │
 └───────────────┘    └───────────────┘    └───────────────┘    └───────────────┘

1. Applications should be either instrumented to generate observability data, or there should be a way to extract observability data from the network or OS kernel level.

2. The collector is an optional element that is either listening or pulling data from applications for further transformation and transferring it to the storage system.

3. As we are dealing with transactional data, data storage is most commonly a time-series database such as Prometheus, InfluxDB or Tempo.

4. UI is used to query and visualize data.

Application Instrumentation

Instrumentation is a fancy way of saying “enabling measurement tools”. OpenTelemetry supports instrumentation for almost all popular programming languages. For a full list, please refer to the documentation page.

Manual Instrumentation

Manual instrumentation is the act of adding observability code to an app yourself. This way is the most labor-intensive because you would need to install the OpenTelemetry SDK, initialize it at the application start, and add metrics, traces, and logs providers in your source code.

The official documentation has numerous examples for all supported programming languages; please refer to this page.

Automatic Instrumentation

Automatic instrumentation aims to reduce the number of code and environment changes required. The underlying mechanism of automatic instrumentation depends on the programming language used. Some mechanisms include monkey patching, bytecode manipulation, or BPF kernel functions. However, automatic instrumentation is available only for a few languages and has some limitations:

Injection of Automatic Instrumentation

If your applications are deployed onto a Kubernetes cluster, you can explore the injection of automatic instrumentation. To do this, you would need to install the OpenTelemetry Operator for Kubernetes. The OpenTelemetry Operator supports injecting and configuring auto-instrumentation libraries for .NET, Java, Node.js, and Python services. For more information, check out the official documentation page.

Collector

The OpenTelemetry collector is a vendor-agnostic way to receive, process, and export telemetry data. It supports receiving data in both the latest supported OpenTelemetry OTLP protocol format as well as in legacy formats or formats from your vendors (such as Jaeger or Zipkin).

Data Storage and UI

Currently, there are numerous enterprise and open-source projects that assist in storing, querying, and visualizing observability data. The table below highlights the most significant projects in this space: