Kotlin Coroutines and OpenTelemetry Tracing – DZone – Uplaza

I lately in contrast three OpenTelemetry approaches on the JVM: Java Agent v1, v2, and Micrometer. I used Kotlin and coroutines with out overthinking. I acquired attention-grabbing suggestions on the utilization of @WithSpan with coroutines:

Certainly, the @WithSpan annotation has labored flawlessly together with coroutines for a while already. Nonetheless, it made me take into consideration the underlying workings of OpenTelemetry. Listed below are my findings.

The @WithSpan Annotation Processor

@WithSpan is an easy annotation. To be of any use, one wants an annotation processor. If you happen to want a refresher on annotation processors, please examine this not-so-new however still-relevant publish.

A fast search on the OpenTelemetry repository reveals that the processor concerned is WithSpanInstrumentation.

Here is an abridged abstract of the lessons concerned:

WithSpanInstrumentation does the annotation processing half; it delegates to WithSpanSingleton. In flip, the latter bridges the decision to the Instrumenter class. Instrumenter accommodates the core of making spans and interacting with the OpenTelemetry collector.

Instrumenter and Context

The Instrumenter encapsulates the complete logic for gathering telemetry, from amassing the information, to beginning and ending spans, to recording values utilizing metrics devices.

An Instrumenter is named in the beginning and the tip of a request/response lifecycle.
When instrumenting a library, there’ll usually be 4 steps.

1. Create an Instrumenter utilizing InstrumenterBuilder. Use the builder to
   configure any library-specific customizations, and likewise expose helpful knobs to your person.
2. Name Instrumenter#shouldStart(Context, Object) and don’t proceed if it returns
   false.
3. Name Instrumenter#begin(Context, Object) at first of a request.
4. Name Instrumenter#finish(Context, Object, Object, Throwable) on the finish of a request.

For extra detailed details about utilizing the Instrumenter see the Utilizing the Instrumenter API web page.

– Instrumenter class

Instrumenter works together with Context. OpenTelemetry API customers must be aware of it, particularly the decision to Context.present(). Let’s describe it in additional element.

Context shops information in a ContextStorage occasion, whose default is ThreadLocal. The ThreadLocal class has been the old-age solution to move information round with out interfering with technique signatures. It shops information within the present thread.

Kotlin’s OpenTelemetry Extension

ThreadLocal works completely — till you spawn different threads. On this case, it’s essential to explicitly move information round. So-called Reactive Programming frameworks, comparable to Spring WebFlux, do spawn different threads; most, if not all, present utilities to deal with the passing robotically.

Coroutines implement Reactive Programming. Not solely do they spawn threads, however additionally they decouple coroutine from threads. A coroutine might “jump” throughout a number of threads in its lifetime. Thus, storing the OpenTelemetry context in a ThreadLocal does not work.

But, coroutines present a devoted storage mechanism, the coroutine context. We’d like a solution to transfer the OpenTelemetry context from the ThreadLocal to the coroutine context and again once more. The best way exists within the opentelemetry-extension-kotlin jar:

The one half that must be added is the place these features are referred to as. Unsurprisingly, the magic occurs within the Java Agent and all different instrumentation lessons. You may bear in mind the TypeInstrumentation interface on the primary diagram, which the category WithSpanInstrumentation carried out. The Java Agent caters to many alternative frameworks and libraries, e.g., Spring WebFlux, and Kotlin Coroutines. Its builders designed it so every TypeInstrumentation concrete class focuses on the instrumentation of a particular side of the framework or library; coroutines are not any exception.

Word that the code supplies a extra particular instrumentation of WithSpanInstrumentation, which is devoted to coroutines.

It seems that the KotlinCoroutinesInstrumentationHelper accommodates the magic to repeat the context from the ThreadLocal to the coroutine context:

bundle io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines;

import io.opentelemetry.context.Context;
import io.opentelemetry.extension.kotlin.ContextExtensionsKt;
import kotlin.coroutines.CoroutineContext;

public last class KotlinCoroutinesInstrumentationHelper {

  public static CoroutineContext addOpenTelemetryContext(CoroutineContext coroutineContext) {
    Context present = Context.present();                                                      //1
    Context inCoroutine = ContextExtensionsKt.getOpenTelemetryContext(coroutineContext);
    if (present == inCoroutine || inCoroutine != Context.root()) {
      return coroutineContext;
    }
    return coroutineContext.plus(ContextExtensionsKt.asContextElement(present));              //2
  }

  personal KotlinCoroutinesInstrumentationHelper() {}
}

1. Get the OpenTelemetry context – from the ThreadLocal. 
2. Add the context to the coroutine context.

And that is a wrap.

Abstract

On this publish, I’ve analyzed the workings of @WithSpan usually and within the context of Kotlin Coroutines. The Java Agent supplies many alternative instrumenting lessons, every devoted to a singular side of a framework or library. The WithSpanInstrumentation within the io.opentelemetry.javaagent.instrumentation.extensionannotations manages “regular” code; the one in io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines manages coroutines.

The largest problem is that OpenTelemetry shops information in a ThreadLocal by default. The coroutine library does not assure the identical thread will likely be used. Quite the opposite, a coroutine will probably bounce throughout completely different threads throughout its lifetime.

The Java Agent supplies the mechanism to deal with it. One half focuses on shifting OpenTelemetry information from the ThreadLocal to the coroutine context; the opposite supplies a devoted instrumentation to name the above code when it enters the latter.

To Go Additional