What’s new in Fluentbit v1.8.1?

by | 17.07.2021 | Changelog

Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below.

New Metrics Support

We have seen that for a long time Fluentbit community have asked for native metrics support. Despite Fluentbit having metrics collectors for CPU, Disk I/O, Network, Memory, and others, we see handling the data payload as a simple structured log called ‘logs as metrics’. Now, we can see the introduction of ‘native metrics’.

For Fluent Bit and Fluentd projects, we know that getting into metrics is exciting, and we understand that the [vendor-neutral](https://www.gartner.com/en/information-technology/glossary/vendor-neutral#:~:text=Vendor-neutral describes a state,or distribution of a specification.&text=Vendor-neutral specifications must be,licensed at a nominal fee.) approach has always been to integrate to what the industry is using as a standard. Hence, as a result of the new version, the team approached Prometheus and OpenMetrics first from an integration point of view.

The new version comes with further Metrics support. It has been a considerable effort to implement several components. The initial implementation consists of the following:

Node Exporter Metrics (Linux only)

The new input plugin called Node Exporter Metrics is a small copy of the great Prometheus Node Exporter. This plugin implements a sub-set of the available metrics but respects all the namespaces and metrics names. So, we can collect your host metrics as you usually do with Prometheus Node Exporter, but this is built-in functionality. The only constraint it contains is that it serves Linux only.

Prometheus Exporter

A new output plugin is available with this new version called Prometheus Exporter. It creates an HTTP endpoint to expose metrics payloads in Prometheus format to scrape any metric.

Again, in this new update, we can see the development of the new library, CMetrics, which will handle all the metrics needs such as counters, gauges and shortly histograms (the last one is currently under development). In addition, we can see some output plugins now are aware of metrics payloads such as stdout, file and influxdb.

Linux Node Metrics to Stdout:

The following code will now help in getting started with linux node metrics to stdout.

docker run -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o stdout -f 1

Linux Node Metrics to Prometheus Exporter

We can see a similar example than above, but now we switch to the Prometheus Exporter output plugin, which exposes metrics on port 2021:

docker run -p 2021:2021 -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o prometheus_exporter -f 1

Next, we have to query the metrics with the command curl:

curl [<http://127.0.0.1:2021/metrics>](<http://127.0.0.1:2021/metrics>)

Upcoming Development on Metrics

During the v1.8.x release cycle, the devs will continue extending the unit metrics reported by Node Exporter Metrics and will publish a new Prometheus Remote Write output plugin and add other functionalities for metrics handling.

New Multiline Core Feature

Multiline has always been a complex problem in production environments. In this release, we can see the implementation of new core functionality to deal with most Multiline use cases. Now, we can ship built-in support for containerized environments mainly based on Docker or CRI plus stacktrace detectors. These environments will be for different languages such as Ruby, Go, Java, etc.

The first plugin which will consume this feature is the Tail input plugin. Without adding any breaking compatibility, we can see that this new version has added a new option to tail to enable this functionality. We can see the simplification of everything by using codes like this:

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri

With the new release, the option multiline.parser uses the new multiline core feature. We can quickly deprecate the old docker mode that allows multi-detection in the format and auto-concatenating messages. Besides built-in multiline parsers, the latest version allows now to register our multiline parsers in the current parsers.conf files with the use of the new [MULTILINE_PARSER] section.

Enhanced Splunk Output Connector

The new version sees the extension of the capabilities of our Splunk connector. With the update, it now supports additional metadata, fields and proper setup and raw mode behaviour.

New HTTP Health end point

We can now get more accurate information about how healthy Fluent Bit is using the new custom health endpoint /api/v1/health. It now calculates error and success rates to indicate an excellent health status.

Kubernetes Filter

The new update shows that the Kubernetes filters use an internal cache to store Pods metadata for avoiding the generation of load in the API Server. With this version, we see a new option to enforce a cache TTL so the cached data can expire at specific intervals. Also, we can use the new option kube_meta_cache_ttl to take advantage of this (disabled by default).

Conclusion

Fluentbit v1.8.1 is finally out now. Try it out here.

Join the Community

The DevOps Awareness Program

Subscribe to the newsletter

Join 100+ cloud native ethusiasts

#wearep3r

Join the community Slack

Discuss all things Kubernetes, DevOps and Cloud Native

More stories from our blog

How to Install Portainer on Remote Server ft. VSCode?

How to Install Portainer on Remote Server ft. VSCode?

Portainer is one of the most popular and trusted GUI for managing Docker, Swarms, ACIs and Kubernetes. The company boasts on its’ website for having 500K users, and there’s no doubt to the number looking at how easy it makes managing the tools. This post goes on the...

What’s new in Python-Tuf v0.18.0?

What’s new in Python-Tuf v0.18.0?

Python-Tuf v0.18.0 recently came, and it is quite a big update with major and minor changes. We will go through all of those changes, additions, fixes and removals in this document. Without further a due, let's start! What is Python-Tuf? The Update Framework (TUF) or...

What’s new in Envoyproxy v1.19.1?

What’s new in Envoyproxy v1.19.1?

Envoyproxy came with its new version a few days ago. Version 1.19.1 comes with very few updates. It provides a few minor behavioural changes and a few bug fixes to make the user experience smoother. In this article, we will cover all of the new changes. Let's start!...

What’s new in Jaeger v1.26.0?

What’s new in Jaeger v1.26.0?

Jaeger v1.26.0 recently came. It has a few changes in its backend. In this article, we will cover all of this in a straightforward way. We will see all of the fixes and the new features that the devs have added. Let's start! What is Jaeger? Jaeger is a graduated CNCF...

Prometheus: As Simple As Possible

Prometheus: As Simple As Possible

Distributed systems help an organisation absorb countless benefits but at the cost of complexity. With the rise of the adoption of container orchestrators like Kubernetes, a need for monitoring and alerting systems came. One such system is Prometheus which is famous...

Bootstrap K3S Data: For Beginners

Bootstrap K3S Data: For Beginners

For Kubernetes users, handling data management tasks and other analysis needs can become difficult with the inclusion of edge based devices. Internet of Things (IoT) as a whole is designed to complement online services for devices commonly used by people such as air...

What’s new in Ingress-Nginx Controller v1.0.0?

What’s new in Ingress-Nginx Controller v1.0.0?

Ingress-Nginx controller for Kubernetes came with its new release almost a month earlier. I know we are pretty late in documenting this but trust me, this update is pretty big. And in this article, we will see all of the new features and essential bug fixes and...

Getting gRPC Right: An Introduction and Review

Getting gRPC Right: An Introduction and Review

The question of APIs and their best implementation through online websites will always remain a tough nut to crack as the web undergoes scaled changes each year. It’s hard to think that the web was once draped by HTML and PHP alone until CSS and Javascript made...

What’s new in TikV v5.0.4?

What’s new in TikV v5.0.4?

TikV came up with its new release this month. It is a small one, but we can see a couple of improvements and some bug fixes along the way. In this article, we will see all of those and view the recent changes. Let's start! What is TikV? TiKV is a graduate project of...

What’s new in Keptn v0.9.0?

What’s new in Keptn v0.9.0?

Keptn recently came out with its new big update with the release of version 0.9.0. We will see what this update contains in this article, along with all of the new features, bug fixes, enhancements and all. Let's start! What is Keptn? Keptn is simply an event-based...

Interested in what we do? Looking for help? Wanna talk about software strategy?