What’s new in Fluentbit v1.8.1?

Published 17.07.2021

Categories Changelog

Tags CNCF, Fluentbit

Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below.

New Metrics Support

We have seen that for a long time Fluentbit community have asked for native metrics support. Despite Fluentbit having metrics collectors for CPU, Disk I/O, Network, Memory, and others, we see handling the data payload as a simple structured log called ‘logs as metrics’. Now, we can see the introduction of ‘native metrics’.

For Fluent Bit and Fluentd projects, we know that getting into metrics is exciting, and we understand that the [vendor-neutral](https://www.gartner.com/en/information-technology/glossary/vendor-neutral#:~:text=Vendor-neutral describes a state,or distribution of a specification.&text=Vendor-neutral specifications must be,licensed at a nominal fee.) approach has always been to integrate to what the industry is using as a standard. Hence, as a result of the new version, the team approached Prometheus and OpenMetrics first from an integration point of view.

The new version comes with further Metrics support. It has been a considerable effort to implement several components. The initial implementation consists of the following:

Node Exporter Metrics (Linux only)

The new input plugin called Node Exporter Metrics is a small copy of the great Prometheus Node Exporter. This plugin implements a sub-set of the available metrics but respects all the namespaces and metrics names. So, we can collect your host metrics as you usually do with Prometheus Node Exporter, but this is built-in functionality. The only constraint it contains is that it serves Linux only.

Prometheus Exporter

A new output plugin is available with this new version called Prometheus Exporter. It creates an HTTP endpoint to expose metrics payloads in Prometheus format to scrape any metric.

Again, in this new update, we can see the development of the new library, CMetrics, which will handle all the metrics needs such as counters, gauges and shortly histograms (the last one is currently under development). In addition, we can see some output plugins now are aware of metrics payloads such as stdout, file and influxdb.

Linux Node Metrics to Stdout:

The following code will now help in getting started with linux node metrics to stdout.

docker run -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o stdout -f 1

Linux Node Metrics to Prometheus Exporter

We can see a similar example than above, but now we switch to the Prometheus Exporter output plugin, which exposes metrics on port 2021:

docker run -p 2021:2021 -v /proc:/host/proc -v /sys:/host/sys    \\
    fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\
        -i node_exporter_metrics      \\
            -p scrape_interval=2      \\
            -p path.procfs=/host/proc \\
            -p path.sysfs=/host/sys   \\
            -o prometheus_exporter -f 1

Next, we have to query the metrics with the command curl:

curl [<http://127.0.0.1:2021/metrics>](<http://127.0.0.1:2021/metrics>)

Upcoming Development on Metrics

During the v1.8.x release cycle, the devs will continue extending the unit metrics reported by Node Exporter Metrics and will publish a new Prometheus Remote Write output plugin and add other functionalities for metrics handling.

New Multiline Core Feature

Multiline has always been a complex problem in production environments. In this release, we can see the implementation of new core functionality to deal with most Multiline use cases. Now, we can ship built-in support for containerized environments mainly based on Docker or CRI plus stacktrace detectors. These environments will be for different languages such as Ruby, Go, Java, etc.

The first plugin which will consume this feature is the Tail input plugin. Without adding any breaking compatibility, we can see that this new version has added a new option to tail to enable this functionality. We can see the simplification of everything by using codes like this:

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri

With the new release, the option multiline.parser uses the new multiline core feature. We can quickly deprecate the old docker mode that allows multi-detection in the format and auto-concatenating messages. Besides built-in multiline parsers, the latest version allows now to register our multiline parsers in the current parsers.conf files with the use of the new [MULTILINE_PARSER] section.

Enhanced Splunk Output Connector

The new version sees the extension of the capabilities of our Splunk connector. With the update, it now supports additional metadata, fields and proper setup and raw mode behaviour.

New HTTP Health end point

We can now get more accurate information about how healthy Fluent Bit is using the new custom health endpoint /api/v1/health. It now calculates error and success rates to indicate an excellent health status.

Kubernetes Filter

The new update shows that the Kubernetes filters use an internal cache to store Pods metadata for avoiding the generation of load in the API Server. With this version, we see a new option to enforce a cache TTL so the cached data can expire at specific intervals. Also, we can use the new option kube_meta_cache_ttl to take advantage of this (disabled by default).

Conclusion

Fluentbit v1.8.1 is finally out now. Try it out here.

Join 100+ cloud native enthusiasts

and stay in the loop on modern software development.

Sign up to receive exclusive content around cloud native software development right into your inbox.

We don’t spam! Read our privacy policy for more info.

More stories from our blog

What’s new in Harbor v2.3.0?

What’s new in Harbor v2.3.0?

Harbor v2.3.0 is here, and with all recent versions, it has brought several new features and has undergone some changes. Let's roll! New Features Declarative Configuration In the latest release, we can see the enhancement of the current configuration by adding...

What’s new in Kubernetes v1.21.2?

What’s new in Kubernetes v1.21.2?

It's June, and Kubernetes has released a new update with version 1.21.2. We will have a look in brief at the changes that came along with this update. We will also have a look at the bugs that Kubernetes removed ahead with the few things added. Let's roll. Changes...

Chaos Engineering: Not so Chaotic

Chaos Engineering: Not so Chaotic

It feels very complex when we talk a lot about cloud computing and developer operations. Furthermore, certain things look complicated, but they are not so if we easily understand those concepts. Today, we will discuss such a thing that sounds complex but is simple and...

On Charming Engineering Culture: My Notes

On Charming Engineering Culture: My Notes

Engineering teams are at the core of any modern organisation. They break/make an organisation, and empowering them is critical to any modern companies’ success. A motivated engineer brings more value than a ‘whatever’ engineer. Its high time managers and leaders focus...

Observability: Your Eyes in Cloud

Observability: Your Eyes in Cloud

Observability is all around the cloud. You might come across the term while exploring the vast stretches of documentations or blog posts, maybe videos or streams too. Well, from far you might have seen that this is a very broad term, and it’s expected. The topic is...

Cloud Firewalls Simplified: Beginners  Edition

Cloud Firewalls Simplified: Beginners Edition

Cloud technology is everywhere. From your photos to big corporations carrying out their day to day operations. But have you ever thought about the security needed to protect this vast pile of data? Security from external attacks by threat detection and elimination is...

Object and Block Storage: How They Differ?

Object and Block Storage: How They Differ?

The difference between block and file storage makes heads spin due to the complexity of definitions and technical jargon across the internet. Even a technical person sometimes forgets the business value and makes decision fatigue their best friend when trying to...

Helm: Why DevOps Engineers Love it?

Helm: Why DevOps Engineers Love it?

Kubernetes doesn’t have reproducibility built-in. At least, that’s what we hear most people complain as a cloud native consultation firm serving both startups and enterprises. I have been using Kubernetes for a while now, and it stands up to the mark of being a gold...

Interested in what we do? Looking for help? Wanna talk about software strategy?