Are you prepared to begin your journey on the highway to accumulating telemetry knowledge out of your purposes? Nice observability begins with nice instrumentation!
On this sequence, you will discover learn how to undertake OpenTelemetry (OTel) and learn how to instrument an utility to gather tracing telemetry. You may discover ways to leverage out-of-the-box computerized instrumentation instruments and perceive when it is necessary to discover extra superior guide instrumentation to your purposes. By the tip of this sequence, you will have an understanding of how telemetry travels out of your purposes to the OpenTelemetry Collector, and be able to deliver OpenTelemetry to your future tasks. Every thing mentioned right here is supported by a hands-on, self-paced workshop authored by Paige Cruz.
The earlier article explored the primary a part of learn how to hyperlink metrics to hint knowledge utilizing exemplars, the place we configured our utility to show metrics and a Prometheus occasion to gather these metrics. On this article, we’ll take a look at the second half centered on implementing the exemplars and tying collectively metrics with our hint knowledge.
It’s assumed that you just adopted the earlier articles in organising each OpenTelemetry and the instance Python utility mission, but when not, return and see the earlier articles as it is not coated right here.
Prometheus Exemplar Configuration
Let’s begin by including exemplars to our Prometheus metrics which is tying in our traces from OpenTelemetry. Open up the metrics/prometheus/prometheus.yml file, add or confirm there’s a scrape job outlined for the hello-otel
utility as proven, and save the file:
international: scrape_interval: 5s scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "hello-otel" static_configs: - targets: ["localhost:5000"]
Open up the metrics/prometheus/Buildfile-prom and make sure the CMD is utilizing the exemplar storage characteristic by including the flag as proven right here in daring:
FROM promenade/prometheus:v2.54.1 ADD prometheus.yml /and so forth/prometheus ENTRYPOINT [ "prometheus" ] CMD [ "--config.file=/etc/prometheus/prometheus.yml", "--enable-feature=exemplar-storage" ]
Utilizing this configuration we are able to now rebuild our Prometheus occasion with:
$ podman construct -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Construct file-prom
Now let’s discover learn how to add metrics to our instance utility.
Software Metrics
Open the metrics/app.py file and make sure the import for the Counter
metric from Prometheus Shopper has been added as proven in daring:
import random import re import urllib3 import requests from flask import Flask, render_template, request from breeds import breeds from opentelemetry.hint import set_tracer_provider from opentelemetry.sdk.hint import TracerProvider from opentelemetry.sdk.hint.export import SimpleSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from prometheus_flask_exporter import PrometheusMetrics from prometheus_client import Counter ...
On this identical utility file, additional down, be sure that a hits counter for the variety of homepage masses is added as proven in daring:
... supplier = TracerProvider() processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")) supplier.add_span_processor(processor) set_tracer_provider(supplier) app = Flask("hello-otel") FlaskInstrumentor().instrument_app(app) Jinja2Instrumentor().instrument() RequestsInstrumentor().instrument() metrics = PrometheusMetrics(app) HITS_COUNTER = Counter('hits_counter', 'rely of homepage masses') ...
Lastly within the utility file, guarantee code is added to the index()
to increment (improve) the hits_counter
and configure exemplars to connect to the metric as proven in daring. Save the file when accomplished:
... @app.route("https://dzone.com/") def index(): international HITS span = hint.get_current_span() trace_id = '{:032x}'.format(span.get_span_context().trace_id) HITS = HITS + 1 span.set_attribute("hits", HITS) HITS_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url": f"http://localhost:16686/trace/{trace_id}"}) msg = f'This webpage has been considered {HITS} occasions' return msg ...
Now we have to replace our utility container picture by including the command to put in prometheus_flask_exporter
to the file metrics/Buildfile-metrics as proven in daring:
FROM python:3.12-bullseye WORKDIR /app COPY necessities.txt necessities.txt RUN pip set up -r necessities.txt RUN pip set up opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-jinja2 opentelemetry-instrumentation-requests prometheus-flask-exporter COPY . . CMD [ "flask", "run", "--host=0.0.0.0"]
Rebuild the applying picture as follows:
$ podman construct -t hello-otel:metrics -f metrics/Buildfile-metrics ... Efficiently tagged localhost/hello-otel:metrics 81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f
Now we have to confirm that this all works.
Verifying Exemplars
Now to confirm our exemplars, we run a pod configuration with our instance utility, a Prometheus occasion, and the Jaeger tooling to visualise our telemetry knowledge as follows:
$ podman play kube metrics/app_pod.yaml
As soon as this has began, we are able to open a browser and make a number of requests (over time, refresh the browser to generate extra within the ensuing graph on the subsequent slide) to the homepage to generate metrics and traces at http://localhost:8001.
Open the Prometheus console in your browser http://localhost:9090 and question hits_counter_total
in Graph view. Choose by clicking on the Present Exemplars button within the center to see them as little blue diamonds (word it’s essential to scale back the time window down to five minutes or so to see a graph plot):
Click on on one of many exemplars (any of the blue dots) on the chart and replica the trace_url
into a brand new browser tab as proven:
If all is working as we anticipate it to, then we should always see the hint waterfall for one of many requests made one thing like this:
This verifies that we’re linking our utility metrics to our utility traces with exemplars. To be neat about issues, we should always shut down our pod and operating containers as follows:
$ podman play kube metrics/app_pod.yaml --down
These examples use code from a Python utility which you can discover within the offered hands-on workshop.
What’s Subsequent?
This text, half two, concludes the journey to linking metrics to our hint knowledge with exemplars utilizing Prometheus and OpenTelemetry with our instance utility.
Keep tuned for extra hands-on materials that can assist you together with your cloud-native observability journey.