I lately in contrast three OpenTelemetry approaches on the JVM: Java Agent v1, v2, and Micrometer. I used Kotlin and coroutines with out overthinking. I acquired attention-grabbing suggestions on the utilization of @WithSpan
with coroutines:
Certainly, the @WithSpan
annotation has labored flawlessly together with coroutines for a while already. Nonetheless, it made me take into consideration the underlying workings of OpenTelemetry. Listed below are my findings.
The @WithSpan Annotation Processor
@WithSpan
is an easy annotation. To be of any use, one wants an annotation processor. If you happen to want a refresher on annotation processors, please examine this not-so-new however still-relevant publish.
A fast search on the OpenTelemetry repository reveals that the processor concerned is WithSpanInstrumentation
.
Here is an abridged abstract of the lessons concerned:
WithSpanInstrumentation
does the annotation processing half; it delegates to WithSpanSingleton
. In flip, the latter bridges the decision to the Instrumenter
class. Instrumenter
accommodates the core of making spans and interacting with the OpenTelemetry collector.
Instrumenter and Context
The
Instrumenter
encapsulates the complete logic for gathering telemetry, from amassing the information, to beginning and ending spans, to recording values utilizing metrics devices.An
Instrumenter
is named in the beginning and the tip of a request/response lifecycle.
When instrumenting a library, there’ll usually be 4 steps.
- Create an
Instrumenter
utilizingInstrumenterBuilder
. Use the builder to
configure any library-specific customizations, and likewise expose helpful knobs to your person.- Name
Instrumenter#shouldStart(Context, Object)
and don’t proceed if it returnsfalse
.- Name
Instrumenter#begin(Context, Object)
at first of a request.- Name
Instrumenter#finish(Context, Object, Object, Throwable)
on the finish of a request.For extra detailed details about utilizing the
Instrumenter
see the Utilizing the Instrumenter API web page.– Instrumenter class
Instrumenter
works together with Context
. OpenTelemetry API customers must be aware of it, particularly the decision to Context.present()
. Let’s describe it in additional element.
Context
shops information in a ContextStorage
occasion, whose default is ThreadLocal
. The ThreadLocal
class has been the old-age solution to move information round with out interfering with technique signatures. It shops information within the present thread.
Kotlin’s OpenTelemetry Extension
ThreadLocal
works completely — till you spawn different threads. On this case, it’s essential to explicitly move information round. So-called Reactive Programming frameworks, comparable to Spring WebFlux, do spawn different threads; most, if not all, present utilities to deal with the passing robotically.
Coroutines implement Reactive Programming. Not solely do they spawn threads, however additionally they decouple coroutine from threads. A coroutine might “jump” throughout a number of threads in its lifetime. Thus, storing the OpenTelemetry context in a ThreadLocal
does not work.
But, coroutines present a devoted storage mechanism, the coroutine context. We’d like a solution to transfer the OpenTelemetry context from the ThreadLocal
to the coroutine context and again once more. The best way exists within the opentelemetry-extension-kotlin
jar:
The one half that must be added is the place these features are referred to as. Unsurprisingly, the magic occurs within the Java Agent and all different instrumentation lessons. You may bear in mind the TypeInstrumentation
interface on the primary diagram, which the category WithSpanInstrumentation
carried out. The Java Agent caters to many alternative frameworks and libraries, e.g., Spring WebFlux, and Kotlin Coroutines. Its builders designed it so every TypeInstrumentation
concrete class focuses on the instrumentation of a particular side of the framework or library; coroutines are not any exception.
Word that the code supplies a extra particular instrumentation of WithSpanInstrumentation
, which is devoted to coroutines.
It seems that the KotlinCoroutinesInstrumentationHelper
accommodates the magic to repeat the context from the ThreadLocal
to the coroutine context:
bundle io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines;
import io.opentelemetry.context.Context;
import io.opentelemetry.extension.kotlin.ContextExtensionsKt;
import kotlin.coroutines.CoroutineContext;
public last class KotlinCoroutinesInstrumentationHelper {
public static CoroutineContext addOpenTelemetryContext(CoroutineContext coroutineContext) {
Context present = Context.present(); //1
Context inCoroutine = ContextExtensionsKt.getOpenTelemetryContext(coroutineContext);
if (present == inCoroutine || inCoroutine != Context.root()) {
return coroutineContext;
}
return coroutineContext.plus(ContextExtensionsKt.asContextElement(present)); //2
}
personal KotlinCoroutinesInstrumentationHelper() {}
}
- Get the OpenTelemetry context – from the
ThreadLocal
. - Add the context to the coroutine context.
And that is a wrap.
Abstract
On this publish, I’ve analyzed the workings of @WithSpan
usually and within the context of Kotlin Coroutines. The Java Agent supplies many alternative instrumenting lessons, every devoted to a singular side of a framework or library. The WithSpanInstrumentation
within the io.opentelemetry.javaagent.instrumentation.extensionannotations
manages “regular” code; the one in io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines
manages coroutines.
The largest problem is that OpenTelemetry shops information in a ThreadLocal
by default. The coroutine library does not assure the identical thread will likely be used. Quite the opposite, a coroutine will probably bounce throughout completely different threads throughout its lifetime.
The Java Agent supplies the mechanism to deal with it. One half focuses on shifting OpenTelemetry information from the ThreadLocal
to the coroutine context; the opposite supplies a devoted instrumentation to name the above code when it enters the latter.