Fluentbit v1.8.1

What’s new in Fluentbit v1.8.1?

by | 17.07.2021 | Changelog

Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below.

New Metrics Support

We have seen that for a long time Fluentbit community have asked for native metrics support. Despite Fluentbit having metrics collectors for CPU, Disk I/O, Network, Memory, and others, we see handling the data payload as a simple structured log called ‘logs as metrics’. Now, we can see the introduction of ‘native metrics’.

For Fluent Bit and Fluentd projects, we know that getting into metrics is exciting, and we understand that the [vendor-neutral](https://www.gartner.com/en/information-technology/glossary/vendor-neutral#:~:text=Vendor-neutral describes a state,or distribution of a specification.&text=Vendor-neutral specifications must be,licensed at a nominal fee.) approach has always been to integrate to what the industry is using as a standard. Hence, as a result of the new version, the team approached Prometheus and OpenMetrics first from an integration point of view.

The new version comes with further Metrics support. It has been a considerable effort to implement several components. The initial implementation consists of the following:

Node Exporter Metrics (Linux only)

The new input plugin called Node Exporter Metrics is a small copy of the great Prometheus Node Exporter. This plugin implements a sub-set of the available metrics but respects all the namespaces and metrics names. So, we can collect your host metrics as you usually do with Prometheus Node Exporter, but this is built-in functionality. The only constraint it contains is that it serves Linux only.

Prometheus Exporter

A new output plugin is available with this new version called Prometheus Exporter. It creates an HTTP endpoint to expose metrics payloads in Prometheus format to scrape any metric.

Again, in this new update, we can see the development of the new library, CMetrics, which will handle all the metrics needs such as counters, gauges and shortly histograms (the last one is currently under development). In addition, we can see some output plugins now are aware of metrics payloads such as stdout, file and influxdb.

Linux Node Metrics to Stdout:

The following code will now help in getting started with linux node metrics to stdout.

docker run -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o stdout -f 1

Linux Node Metrics to Prometheus Exporter

We can see a similar example than above, but now we switch to the Prometheus Exporter output plugin, which exposes metrics on port 2021:

docker run -p 2021:2021 -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o prometheus_exporter -f 1

Next, we have to query the metrics with the command curl:

curl [<http://127.0.0.1:2021/metrics>](<http://127.0.0.1:2021/metrics>)

Upcoming Development on Metrics

During the v1.8.x release cycle, the devs will continue extending the unit metrics reported by Node Exporter Metrics and will publish a new Prometheus Remote Write output plugin and add other functionalities for metrics handling.

New Multiline Core Feature

Multiline has always been a complex problem in production environments. In this release, we can see the implementation of new core functionality to deal with most Multiline use cases. Now, we can ship built-in support for containerized environments mainly based on Docker or CRI plus stacktrace detectors. These environments will be for different languages such as Ruby, Go, Java, etc.

The first plugin which will consume this feature is the Tail input plugin. Without adding any breaking compatibility, we can see that this new version has added a new option to tail to enable this functionality. We can see the simplification of everything by using codes like this:

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri

With the new release, the option multiline.parser uses the new multiline core feature. We can quickly deprecate the old docker mode that allows multi-detection in the format and auto-concatenating messages. Besides built-in multiline parsers, the latest version allows now to register our multiline parsers in the current parsers.conf files with the use of the new [MULTILINE_PARSER] section.

Enhanced Splunk Output Connector

The new version sees the extension of the capabilities of our Splunk connector. With the update, it now supports additional metadata, fields and proper setup and raw mode behaviour.

New HTTP Health end point

We can now get more accurate information about how healthy Fluent Bit is using the new custom health endpoint /api/v1/health. It now calculates error and success rates to indicate an excellent health status.

Kubernetes Filter

The new update shows that the Kubernetes filters use an internal cache to store Pods metadata for avoiding the generation of load in the API Server. With this version, we see a new option to enforce a cache TTL so the cached data can expire at specific intervals. Also, we can use the new option kube_meta_cache_ttl to take advantage of this (disabled by default).

Conclusion

Fluentbit v1.8.1 is finally out now. Try it out here.

CommunityNew

The DevOps Awareness Program

Subscribe to the newsletter

Join 100+ cloud native ethusiasts

#wearep3r

Join the community Slack

Discuss all things Kubernetes, DevOps and Cloud Native

Related articles6

What’s new in Kuma v1.3.0?

What’s new in Kuma v1.3.0?

Kuma recently came with their new version of 1.3.0. It has come up with several bug fixes and new features with this update. In this article, we will see those fixes and new features which will make users have a great experience with the product. Buck up, and let’s...

What’s new in Istio v1.11.3?

What’s new in Istio v1.11.3?

Istio came with its new version recently. It is a minor release, but it contains some significant changes and fixes. In this article, we will have a detailed look at what version 1.11.3 brings to the table. So, without wasting any time. Let's start! What is Istio?...

What’s new in Traefik v2.5.3?

What’s new in Traefik v2.5.3?

Traefik came with a new version of 2.5.3. This version mainly focuses on bug fixing and adding documents. This article will cover all of those entirely. It is not a big update, so this article will be short and crisp. Buckle up for a ride. Let's start! What is...

What’s new in Prometheus v2.30?

What’s new in Prometheus v2.30?

Prometheus v2.30 was released a few days ago, and it is an exciting update. This update is not very inclined on adding new features to the ecosystem, but it brings several enhancements to configurability and resource usage efficiency. It also brings several bug fixes....

What’s new in Python-Tuf v0.18.0?

What’s new in Python-Tuf v0.18.0?

Python-Tuf v0.18.0 recently came, and it is quite a big update with major and minor changes. We will go through all of those changes, additions, fixes and removals in this document. Without further a due, let's start! What is Python-Tuf? The Update Framework (TUF) or...

What’s new in Envoyproxy v1.19.1?

What’s new in Envoyproxy v1.19.1?

Envoyproxy came with its new version a few days ago. Version 1.19.1 comes with very few updates. It provides a few minor behavioural changes and a few bug fixes to make the user experience smoother. In this article, we will cover all of the new changes. Let's start!...