Linking Metrics to Traces With Exemplars, Half 2 – DZone – Uplaza

Are you prepared to begin your journey on the highway to accumulating telemetry knowledge out of your purposes? Nice observability begins with nice instrumentation! 

On this sequence, you will discover learn how to undertake OpenTelemetry (OTel) and learn how to instrument an utility to gather tracing telemetry. You may discover ways to leverage out-of-the-box computerized instrumentation instruments and perceive when it is necessary to discover extra superior guide instrumentation to your purposes. By the tip of this sequence, you will have an understanding of how telemetry travels out of your purposes to the OpenTelemetry Collector, and be able to deliver OpenTelemetry to your future tasks. Every thing mentioned right here is supported by a hands-on, self-paced workshop authored by Paige Cruz. 

The earlier article explored the primary a part of learn how to hyperlink metrics to hint knowledge utilizing exemplars, the place we configured our utility to show metrics and a Prometheus occasion to gather these metrics. On this article, we’ll take a look at the second half centered on implementing the exemplars and tying collectively metrics with our hint knowledge.

It’s assumed that you just adopted the earlier articles in organising each OpenTelemetry and the instance Python utility mission, but when not, return and see the earlier articles as it is not coated right here.

Prometheus Exemplar Configuration

Let’s begin by including exemplars to our Prometheus metrics which is tying in our traces from OpenTelemetry. Open up the metrics/prometheus/prometheus.yml file, add or confirm there’s a scrape job outlined for the hello-otel utility as proven, and save the file:

international:
  scrape_interval: 5s

scrape_configs:

  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "hello-otel"
    static_configs:
      - targets: ["localhost:5000"]

Open up the metrics/prometheus/Buildfile-prom and make sure the CMD is utilizing the exemplar storage characteristic by including the flag as proven right here in daring:

FROM promenade/prometheus:v2.54.1

ADD prometheus.yml /and so forth/prometheus

ENTRYPOINT [ "prometheus" ]

CMD [ "--config.file=/etc/prometheus/prometheus.yml", "--enable-feature=exemplar-storage" ]

Utilizing this configuration we are able to now rebuild our Prometheus occasion with:

$ podman construct -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Construct file-prom

Now let’s discover learn how to add metrics to our instance utility.

Software Metrics

Open the metrics/app.py file and make sure the import for the Counter metric from Prometheus Shopper has been added as proven in daring:

import random
import re
import urllib3

import requests
from flask import Flask, render_template, request
from breeds import breeds

from opentelemetry.hint import set_tracer_provider
from opentelemetry.sdk.hint import TracerProvider
from opentelemetry.sdk.hint.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from prometheus_flask_exporter import PrometheusMetrics
from prometheus_client import Counter
...

On this identical utility file, additional down, be sure that a hits counter for the variety of homepage masses is added as proven in daring:

...
supplier = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
supplier.add_span_processor(processor)

set_tracer_provider(supplier)

app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)

HITS_COUNTER = Counter('hits_counter', 'rely of homepage masses')
...

Lastly within the utility file, guarantee code is added to the index() to increment (improve) the hits_counter and configure exemplars to connect to the metric as proven in daring. Save the file when accomplished:

...
@app.route("https://dzone.com/")
def index():
  international HITS
  span = hint.get_current_span()
  trace_id = '{:032x}'.format(span.get_span_context().trace_id)
  HITS = HITS + 1
  span.set_attribute("hits", HITS)
  HITS_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url": f"http://localhost:16686/trace/{trace_id}"})
  msg = f'This webpage has been considered {HITS} occasions'
  return msg
...

Now we have to replace our utility container picture by including the command to put in prometheus_flask_exporter to the file metrics/Buildfile-metrics as proven in daring:

FROM python:3.12-bullseye

WORKDIR /app

COPY necessities.txt necessities.txt

RUN pip set up -r necessities.txt

RUN pip set up opentelemetry-api 
    opentelemetry-sdk 
    opentelemetry-exporter-otlp 
    opentelemetry-instrumentation-flask 
    opentelemetry-instrumentation-jinja2 
    opentelemetry-instrumentation-requests 
    prometheus-flask-exporter

COPY . .

CMD [ "flask", "run", "--host=0.0.0.0"]

Rebuild the applying picture as follows:

$ podman construct -t hello-otel:metrics -f metrics/Buildfile-metrics

...
Efficiently tagged localhost/hello-otel:metrics
81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f

Now we have to confirm that this all works.

Verifying Exemplars

Now to confirm our exemplars, we run a pod configuration with our instance utility, a Prometheus occasion, and the Jaeger tooling to visualise our telemetry knowledge as follows:

$ podman play kube metrics/app_pod.yaml 

As soon as this has began, we are able to open a browser and make a number of requests (over time, refresh the browser to generate extra within the ensuing graph on the subsequent slide) to the homepage to generate metrics and traces at http://localhost:8001.

Open the Prometheus console in your browser http://localhost:9090 and question hits_counter_total in Graph view. Choose by clicking on the Present Exemplars button within the center to see them as little blue diamonds (word it’s essential to scale back the time window down to five minutes or so to see a graph plot):

Click on on one of many exemplars (any of the blue dots) on the chart and replica the trace_url into a brand new browser tab as proven:

If all is working as we anticipate it to, then we should always see the hint waterfall for one of many requests made one thing like this:

This verifies that we’re linking our utility metrics to our utility traces with exemplars. To be neat about issues, we should always shut down our pod and operating containers as follows:

$ podman play kube metrics/app_pod.yaml --down

These examples use code from a Python utility which you can discover within the offered hands-on workshop.

What’s Subsequent?

This text, half two, concludes the journey to linking metrics to our hint knowledge with exemplars utilizing Prometheus and OpenTelemetry with our instance utility.

Keep tuned for extra hands-on materials that can assist you together with your cloud-native observability journey.