What’s new in Prometheus 2.28?

Published 08.07.2021

Categories Changelog

Prometheus 2.28 is out. If you don’t know, Prometheus is an excellent open-source system monitoring and alerting toolkit. Let’s have a look at those features and have a look at the changelog.

Displaying Trace Examplers in the Graphic Interface

From the previous versions of 2.26 and 2.27, we can see the new support that Prometheus received for storing and receiving trace exemplars and sending exemplar data over Prometheus’s remote_write interface.

In this version, we see that the integration of exemplars with metrics is taking a step further with added support for displaying tracing exemplars in Prometheus’s graphing interface. From now on, whenever a PromQL query accesses series with exemplar data attached to them, we can now show these exemplars along with the returned PromQL query data by enabling a per graph “Show Exemplars” setting.

prometheus 2.28

Per Graph Show Exemplars (Source: Promlabs)

A latency-related graph allows us to quickly find trace exemplars for requests with high latency to investigate those requests in a tracing system further. Suppose we want to copy part of the exemplar metadata (such as the trace ID) to correlate it with another system in production. In that case, we can click on any exemplar to show its metadata more persistently under the graph.

An easy way for us to see this is to get a demo running where we can try out the exemplar display is by running Grafana’s “The New Stack” demo app:

# Clone the "The New Stack" demo app.
git clone git@github.com:grafana/tns

# Check out a predictable version of the repo.
git checkout ec5673d

# Use Prometheus 2.28 instead of 2.26.
cd production/docker-compose
sed -i 's/prometheus:v2.26/prometheus:v2.28/' docker-compose.yml

# Run components necessary to expose and store metrics and trace exemplars.
docker-compose up tempo db app loadgen prometheus

After this, we can head to  http://localhost:9090/ and run the query:

histogram_quantile(0.99, sum by(le, method) (rate(tns_request_duration_seconds_bucket[1m])))

We can click the “Show Exemplars” button to show trace exemplars.

Generic HTTP-Based Service Discovery

We have seen earlier that Prometheus has supported generic service discovery integrations for a long time via its file_sd discovery mechanism, which watches and reads a set of target files from disk. The only downside of the file_sd approach is that it requires sharing a filesystem between Prometheus and a sidecar process that generates the target files, which can be problematic in some environments. Users have asked for a network-based alternative for a long time.

A new HTTP-based generic discovery mechanism is now in Prometheus v2.28. This mechanism works almost like file_sd but loads target information from a specified URL rather than from a local file:

- job_name: http-sd
    - url: 'https://infra-db/prometheus-sd'

It also only supports JSON as the targets format, whereas file_sd supports both JSON and YAML.

This new HTTP-based discovery mechanism not only encourages people to build remote sidecar processes that can pull target information from various sources of truth. But developers will add native Prometheus HTTP discovery support directly into their infrastructure and service databases, getting rid of the need to run a separate process.

Defaulting to the New Expression Editor

During Prometheus 2.26, we saw the introduction of a shiny new PromQL editor with advanced auto-completion, inline linting, and syntax, highlighting that you had to explicitly enable it as an experimental feature. Since this new editor is so much more usable than the previous bare text input field, and since no major complaints came about it, Prometheus 2.28 now defaults to this new editor:

New Editor (Source: Promlabs)

This default is stored in the browser’s local storage and is only set to the new default value if no previously stored value is present for it yet. Thus if we have used Prometheus 2.26 or 2.27 in the meantime and now access 2.28 using the same URL, we may still have to enable the new editor explicitly.

Other Features and Enhancements

Promtool: We will now see an allowance of silencing output when importing or backfilling data.

Consul SD: It will now support reading tokens from the file.

Rules: Added a new .ExternalURL alert field templating variable containing the external URL of the Prometheus server.

Scrape: Added experimental body_size_limit scrape configuration setting to limit the allowed response body size for target scrapes.

Kubernetes SD: Added ingress class name label for ingress discovery.

UI: Showing of a startup screen with a progress bar when the TSDB is not ready yet.

SD: Addition of a target creation failure counter like prometheus_target_sync_failed_total and improvement of target creation failure handling.

TSDB: Improving validation of exemplar label set length.

TSDB: Added a prometheus_tsdb_clean_start metric indicating whether a TSDB lockfile from a previous run still existed upon startup.

Bug Fixes

UI: In the experimental PromQL editor, fixing autocompletion and parsing for unique float values and improving series metadata fetching.

TSDB: When merging chunks, split resulting chunks would contain more than the maximum of 120 samples.

SD: Fixed the computation of the prometheus_sd_discovered_targets metric when using multiple service discoveries.


A lot of new things came with Prometheus v2.28. You can try out this new version by clicking here.

Read more posts on CNCF here:

Join 100+ cloud native enthusiasts

and stay in the loop on modern software development.

Sign up to receive exclusive content around cloud native software development right into your inbox.

We don’t spam! Read our privacy policy for more info.

More stories from our blog

How To Install Docker on Ubuntu 20.04?

How To Install Docker on Ubuntu 20.04?

Docker is an open-source tool that makes managing application processes in containers much easier. Containers allow you to run your programmes in separate processes with their own resources. Containers are comparable to virtual machines, except they're more portable,...

Answer to Everything isn’t 42, it’s Family

Answer to Everything isn’t 42, it’s Family

We’re experiencing digitisation. An era where every person has a voice, and it doesn’t matter if he’s wise. There’s more motivation circulating the vast stretches of the internet than it’s required. This would be good in a theoretical world, but if you seek the truth,...

What’s new in Gitlab 14? 🦊

What’s new in Gitlab 14? 🦊

GitLab 14 is out and fans must be thrilled to know about all the new features along with all the fixes and removals. In this post, we will go through the many changes and improvements, bug fixes, and some remarkable deprecations. We will see all of that here. So,...

k8s vs k3s: The Comprehensive Difference

k8s vs k3s: The Comprehensive Difference

Kubernetes is undoubtedly a champion in the container orchestration world. But currently, we see that K3s or a lightweight Kubernetes distribution which is light, efficient and fast with a drastically small footprint levelling up. Businesses nowadays scratch their...

What’s new in Fluentbit v1.8.1?

What’s new in Fluentbit v1.8.1?

Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below. New Metrics...

What’s new in Envoy v1.19.0?

What’s new in Envoy v1.19.0?

Envoyproxy introduced its new version, 1.19.0, recently, and it came with many changes and improvements from the previous ones. We can see more stability in this version, along with specific bug fixes. So, without waiting any further, let’s see what the new version...

What’s new in Vitess 10?

What’s new in Vitess 10?

Vitess 10 is released with many excellent features and also many bug fixes that were bothering the user base. We are going to see all the features and exciting announcements. So, Let's roll! Major Themes in Vitess In this release, we can see that Vitess Maintainers...

What’s new in Contour 1.17.0?

What’s new in Contour 1.17.0?

Contour 1.17.0 is out with a layer seven HTTP reverse proxy for Kubernetes clusters. The new version has arrived with many new features and several fixes, which will make the functioning of the ingress controller smoother. More activities within the community came...

GPT-J: GPT-3 Democratized

GPT-J: GPT-3 Democratized

GPT-J is the open source cousin of GPT-3 that everyone can use. The open source transformer is all about democratizing transformers and with 6b parameters it’s the largest transformer available. Read more here

What’s new in Portainer CE 2.6.0?

What’s new in Portainer CE 2.6.0?

Portainer fans will be very excited because a new version came out recently. We are going to see the changes that came in this release and also going to explore the features. We will also see the issues and bugs that got fixed to maintain stability and improve user...

Interested in what we do? Looking for help? Wanna talk about software strategy?