Are you prepared to begin your journey on the highway to amassing telemetry information out of your functions? Nice observability begins with nice instrumentation!
On this sequence, you will discover easy methods to undertake OpenTelemetry (OTel) and easy methods to instrument an software to gather tracing telemetry. You will discover ways to leverage out-of-the-box automated instrumentation instruments and perceive when it is necessary to discover extra superior guide instrumentation in your functions. By the top of this sequence, you will have an understanding of how telemetry travels out of your functions to the OpenTelemetry Collector, and be able to convey OpenTelemetry to your future initiatives. All the pieces mentioned right here is supported by a hands-on, self-paced workshop authored by Paige Cruz.
The earlier article explored how builders are capable of manually instrument their functions to generate particular metadata for our enterprise to derive higher insights quicker. On this article, we’ll have a look at the primary a part of easy methods to hyperlink metrics to hint information utilizing exemplars.
It’s assumed that you simply adopted the earlier articles in establishing each OpenTelemetry and the instance Python software challenge, but when not, return and see the earlier articles as it isn’t lined right here.
Earlier than we dive into linking our metrics to traces with exemplars, let us take a look at what precisely an exemplar is.
Exemplar Fundamentals
As outlined by Google Cloud Observability:
“Exemplars are a way to associate arbitrary data with metric data. You can use them to attach non-metric data to measurements. One use of exemplars is to associate trace data with metric data.”
Keep in mind, we use hint information to provide an in depth view of a single request by means of our methods and metrics are used to supply an aggregated methods view. Exemplars are a method to mix the 2, such that after you slender your troubleshooting efforts to a single hint, you possibly can then discover the related metrics that the exemplar supplies. This additionally works from metrics to traces, for instance, permitting you to leap from, “Hey, what’s that weird spike?” on a metric chart on to a hint related to that context in a single click on.
Our aim is tying metrics to traces by way of exemplars, so utilizing the CNCF open-source initiatives, we’ll instrument metrics with Prometheus, which has steady APIs and an in depth ecosystem of instrumented libraries. Whereas OpenTelemetry additionally supplies a metrics SDK, it is marked as combined and underneath lively improvement. The Python SDK doesn’t but help exemplars, so let’s hold our traces instrumented with OpenTelemetry.
The plan we’ll observe in Half One is to first configure a Prometheus occasion to assemble metrics from our instance software, second to instrument our instance software to gather metrics, and verifying that that is amassing metrics.
In Half Two, we’ll then implement an exemplar connecting these metrics to current hint information we have been amassing from our instance software, and eventually, confirm all this work in Jaeger tooling.
Prometheus Configuration
Inside the instance Python software we have beforehand put in, we discover a configuration file for Prometheus in metrics/prometheus/prometheus.yml that appears like this:
world:
scrape_interval: 5s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "hello-otel"
static_configs:
- targets: ["localhost:5000"]
The default use of Prometheus is to observe targets over HTTP and scrape their endpoints. Our preliminary configuration is scraping Prometheus itself because the goal with the endpoint localhost:9090 and on the lookout for hello-intel as a goal on the endpoint localhost:5000.
Utilizing this configuration we will now construct and begin our Prometheus occasion with:
$ podman construct -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Buildfile-prom $ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/and so forth/prometheus/prometheus.yml
We will confirm our targets are configured appropriately by opening the Prometheus console Targets web page at http://localhost:9090/targets, noting that Prometheus is scraping itself and ready for the hello-otel software to return on-line:
For now, cease the Prometheus occasion by utilizing CTRL-C as we’ll later be operating the appliance and Prometheus occasion collectively in a single pod.
Accumulating Metrics
The subsequent step is to begin amassing metrics from our Python instance software. To do that we will discover the prometheus-flask-exporter that gives a straightforward getting began expertise.
We begin by opening the construct file metrics/Buildfile-metrics and including the command to put in the prometheus_flask_exporter as proven right here in daring:
FROM python:3.12-bullseye WORKDIR /app COPY necessities.txt necessities.txt RUN pip set up -r necessities.txt RUN pip set up opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-jinja2 opentelemetry-instrumentation-requests prometheus-flask-exporter COPY . . CMD [ "flask", "run", "--host=0.0.0.0"]
Now to begin utilizing it in our instance software we open the file metrics/app.py and add the prometheus-flask-exporter and specify import particularly PrometheusMetrics as proven in daring beneath:
... from opentelemetry.hint import set_tracer_provider from opentelemetry.sdk.hint import TracerProvider from opentelemetry.sdk.hint.export import SimpleSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from prometheus_flask_exporter import PrometheusMetrics ...
Search additional down this similar file, metrics/app.py, and confirm that the programmatic Flask metric instrumentation has been added as proven in daring right here:
...
supplier = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
supplier.add_span_processor(processor)
set_tracer_provider(supplier)
app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)
...
Now that we have now all of it configured, let’s attempt it out.
Verifying Metrics to Traces
We have to begin verification by (re)constructing the appliance picture as follows:
$ podman construct -t hello-otel:metrics -f metrics/Buildfile-metrics ... Efficiently tagged localhost/hello-otel:metrics 81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f
Now to run this, we’re utilizing a pod configuration with our instance software, a Prometheus occasion, and the Jaeger tooling to visualise our telemetry information as follows:
$ podman play kube metrics/app_pod.yaml --down
As soon as this has began, we will open a browser and make a number of requests to every of the pages listed right here beneath to generate metrics and tracing information:
- http://localhost:8001
- http://localhost:8001/rolldice
- http://localhost:8001/doggo
I discussed beforehand that Prometheus makes use of a pull-model the place a Prometheus server scrapes a /metric endpoint. Let’s examine our instance software’s metric endpoint to confirm that the prometheus-flask-exporter is working by opening http://localhost:8001/metrics in a browser and confirming we’re seeing metrics prefixed with flask_* and python_* as follows:
Now after we examine if the hello-otel software is obtainable to be scrapped by Prometheus by opening the targets web page in Prometheus at http://localhost:9090/targets, we see that each Prometheus and the instance software are within the state of UP (it was DOWN, beforehand):
This concludes Half One. These examples use code from a Python software which you can discover within the offered hands-on workshop.
What’s Subsequent?
This text, half considered one of two, began the journey to linking metrics to our hint information with exemplars utilizing Prometheus and OpenTelemetry with our instance software.
In our subsequent article, half two, we’ll end our journey by implementing an exemplar to hyperlink these collected metrics to their related traces.