What’s new in Fluentbit v1.8.1?

Published 17.07.2021

Categories Changelog

Tags CNCF, Fluentbit

Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below.

New Metrics Support

We have seen that for a long time Fluentbit community have asked for native metrics support. Despite Fluentbit having metrics collectors for CPU, Disk I/O, Network, Memory, and others, we see handling the data payload as a simple structured log called ‘logs as metrics’. Now, we can see the introduction of ‘native metrics’.

For Fluent Bit and Fluentd projects, we know that getting into metrics is exciting, and we understand that the [vendor-neutral](https://www.gartner.com/en/information-technology/glossary/vendor-neutral#:~:text=Vendor-neutral describes a state,or distribution of a specification.&text=Vendor-neutral specifications must be,licensed at a nominal fee.) approach has always been to integrate to what the industry is using as a standard. Hence, as a result of the new version, the team approached Prometheus and OpenMetrics first from an integration point of view.

The new version comes with further Metrics support. It has been a considerable effort to implement several components. The initial implementation consists of the following:

Node Exporter Metrics (Linux only)

The new input plugin called Node Exporter Metrics is a small copy of the great Prometheus Node Exporter. This plugin implements a sub-set of the available metrics but respects all the namespaces and metrics names. So, we can collect your host metrics as you usually do with Prometheus Node Exporter, but this is built-in functionality. The only constraint it contains is that it serves Linux only.

Prometheus Exporter

A new output plugin is available with this new version called Prometheus Exporter. It creates an HTTP endpoint to expose metrics payloads in Prometheus format to scrape any metric.

Again, in this new update, we can see the development of the new library, CMetrics, which will handle all the metrics needs such as counters, gauges and shortly histograms (the last one is currently under development). In addition, we can see some output plugins now are aware of metrics payloads such as stdout, file and influxdb.

Linux Node Metrics to Stdout:

The following code will now help in getting started with linux node metrics to stdout.

docker run -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o stdout -f 1

Linux Node Metrics to Prometheus Exporter

We can see a similar example than above, but now we switch to the Prometheus Exporter output plugin, which exposes metrics on port 2021:

docker run -p 2021:2021 -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o prometheus_exporter -f 1

Next, we have to query the metrics with the command curl:

curl [<http://127.0.0.1:2021/metrics>](<http://127.0.0.1:2021/metrics>)

Upcoming Development on Metrics

During the v1.8.x release cycle, the devs will continue extending the unit metrics reported by Node Exporter Metrics and will publish a new Prometheus Remote Write output plugin and add other functionalities for metrics handling.

New Multiline Core Feature

Multiline has always been a complex problem in production environments. In this release, we can see the implementation of new core functionality to deal with most Multiline use cases. Now, we can ship built-in support for containerized environments mainly based on Docker or CRI plus stacktrace detectors. These environments will be for different languages such as Ruby, Go, Java, etc.

The first plugin which will consume this feature is the Tail input plugin. Without adding any breaking compatibility, we can see that this new version has added a new option to tail to enable this functionality. We can see the simplification of everything by using codes like this:

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri

With the new release, the option multiline.parser uses the new multiline core feature. We can quickly deprecate the old docker mode that allows multi-detection in the format and auto-concatenating messages. Besides built-in multiline parsers, the latest version allows now to register our multiline parsers in the current parsers.conf files with the use of the new [MULTILINE_PARSER] section.

Enhanced Splunk Output Connector

The new version sees the extension of the capabilities of our Splunk connector. With the update, it now supports additional metadata, fields and proper setup and raw mode behaviour.

New HTTP Health end point

We can now get more accurate information about how healthy Fluent Bit is using the new custom health endpoint /api/v1/health. It now calculates error and success rates to indicate an excellent health status.

Kubernetes Filter

The new update shows that the Kubernetes filters use an internal cache to store Pods metadata for avoiding the generation of load in the API Server. With this version, we see a new option to enforce a cache TTL so the cached data can expire at specific intervals. Also, we can use the new option kube_meta_cache_ttl to take advantage of this (disabled by default).

Conclusion

Fluentbit v1.8.1 is finally out now. Try it out here.

Join 100+ cloud native enthusiasts

and stay in the loop on modern software development.

Sign up to receive exclusive content around cloud native software development right into your inbox.

We don’t spam! Read our privacy policy for more info.

More stories from our blog

How To Install Docker on Ubuntu 20.04?

How To Install Docker on Ubuntu 20.04?

Docker is an open-source tool that makes managing application processes in containers much easier. Containers allow you to run your programmes in separate processes with their own resources. Containers are comparable to virtual machines, except they're more portable,...

Answer to Everything isn’t 42, it’s Family

Answer to Everything isn’t 42, it’s Family

We’re experiencing digitisation. An era where every person has a voice, and it doesn’t matter if he’s wise. There’s more motivation circulating the vast stretches of the internet than it’s required. This would be good in a theoretical world, but if you seek the truth,...

What’s new in Gitlab 14? 🦊

What’s new in Gitlab 14? 🦊

GitLab 14 is out and fans must be thrilled to know about all the new features along with all the fixes and removals. In this post, we will go through the many changes and improvements, bug fixes, and some remarkable deprecations. We will see all of that here. So,...

k8s vs k3s: The Comprehensive Difference

k8s vs k3s: The Comprehensive Difference

Kubernetes is undoubtedly a champion in the container orchestration world. But currently, we see that K3s or a lightweight Kubernetes distribution which is light, efficient and fast with a drastically small footprint levelling up. Businesses nowadays scratch their...

What’s new in Envoy v1.19.0?

What’s new in Envoy v1.19.0?

Envoyproxy introduced its new version, 1.19.0, recently, and it came with many changes and improvements from the previous ones. We can see more stability in this version, along with specific bug fixes. So, without waiting any further, let’s see what the new version...

What’s new in Vitess 10?

What’s new in Vitess 10?

Vitess 10 is released with many excellent features and also many bug fixes that were bothering the user base. We are going to see all the features and exciting announcements. So, Let's roll! Major Themes in Vitess In this release, we can see that Vitess Maintainers...

What’s new in Contour 1.17.0?

What’s new in Contour 1.17.0?

Contour 1.17.0 is out with a layer seven HTTP reverse proxy for Kubernetes clusters. The new version has arrived with many new features and several fixes, which will make the functioning of the ingress controller smoother. More activities within the community came...

What’s new in Prometheus 2.28?

What’s new in Prometheus 2.28?

Prometheus 2.28 is out. If you don't know, Prometheus is an excellent open-source system monitoring and alerting toolkit. Let's have a look at those features and have a look at the changelog. Displaying Trace Examplers in the Graphic Interface From the previous...

GPT-J: GPT-3 Democratized

GPT-J: GPT-3 Democratized

GPT-J is the open source cousin of GPT-3 that everyone can use. The open source transformer is all about democratizing transformers and with 6b parameters it’s the largest transformer available. Read more here

What’s new in Portainer CE 2.6.0?

What’s new in Portainer CE 2.6.0?

Portainer fans will be very excited because a new version came out recently. We are going to see the changes that came in this release and also going to explore the features. We will also see the issues and bugs that got fixed to maintain stability and improve user...

Interested in what we do? Looking for help? Wanna talk about software strategy?