Fluentbit is a lightweight and fast data processor and forwarder for Linux, BSD and OSX. And, for Fluentbit fans, there is good news as they have released their new update with lots of new features and fixes. We will have a look at all of them below.
New Metrics Support
We have seen that for a long time Fluentbit community have asked for native metrics support. Despite Fluentbit having metrics collectors for CPU, Disk I/O, Network, Memory, and others, we see handling the data payload as a simple structured log called ‘logs as metrics’. Now, we can see the introduction of ‘native metrics’.
For Fluent Bit and Fluentd projects, we know that getting into metrics is exciting, and we understand that the [vendor-neutral](https://www.gartner.com/en/information-technology/glossary/vendor-neutral#:~:text=Vendor-neutral describes a state,or distribution of a specification.&text=Vendor-neutral specifications must be,licensed at a nominal fee.) approach has always been to integrate to what the industry is using as a standard. Hence, as a result of the new version, the team approached Prometheus and OpenMetrics first from an integration point of view.
The new version comes with further Metrics support. It has been a considerable effort to implement several components. The initial implementation consists of the following:
Node Exporter Metrics (Linux only)
The new input plugin called Node Exporter Metrics is a small copy of the great Prometheus Node Exporter. This plugin implements a sub-set of the available metrics but respects all the namespaces and metrics names. So, we can collect your host metrics as you usually do with Prometheus Node Exporter, but this is built-in functionality. The only constraint it contains is that it serves Linux only.
A new output plugin is available with this new version called Prometheus Exporter. It creates an HTTP endpoint to expose metrics payloads in Prometheus format to scrape any metric.
Again, in this new update, we can see the development of the new library, CMetrics, which will handle all the metrics needs such as counters, gauges and shortly histograms (the last one is currently under development). In addition, we can see some output plugins now are aware of metrics payloads such as stdout, file and influxdb.
Linux Node Metrics to Stdout:
The following code will now help in getting started with linux node metrics to stdout.
docker run -v /proc:/host/proc -v /sys:/host/sys \\ fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\ -i node_exporter_metrics \\ -p scrape_interval=2 \\ -p path.procfs=/host/proc \\ -p path.sysfs=/host/sys \\ -o stdout -f 1
Linux Node Metrics to Prometheus Exporter
We can see a similar example than above, but now we switch to the Prometheus Exporter output plugin, which exposes metrics on port 2021:
docker run -p 2021:2021 -v /proc:/host/proc -v /sys:/host/sys \\ fluent/fluent-bit:1.8 fluent-bit/bin/fluent-bit \\ -i node_exporter_metrics \\ -p scrape_interval=2 \\ -p path.procfs=/host/proc \\ -p path.sysfs=/host/sys \\ -o prometheus_exporter -f 1
Next, we have to query the metrics with the command curl:
Upcoming Development on Metrics
During the v1.8.x release cycle, the devs will continue extending the unit metrics reported by Node Exporter Metrics and will publish a new Prometheus Remote Write output plugin and add other functionalities for metrics handling.
New Multiline Core Feature
Multiline has always been a complex problem in production environments. In this release, we can see the implementation of new core functionality to deal with most Multiline use cases. Now, we can ship built-in support for containerized environments mainly based on Docker or CRI plus stacktrace detectors. These environments will be for different languages such as Ruby, Go, Java, etc.
The first plugin which will consume this feature is the Tail input plugin. Without adding any breaking compatibility, we can see that this new version has added a new option to tail to enable this functionality. We can see the simplification of everything by using codes like this:
[INPUT] name tail path /var/log/containers/*.log multiline.parser docker, cri
With the new release, the option
multiline.parser uses the new multiline core feature. We can quickly deprecate the old docker mode that allows multi-detection in the format and auto-concatenating messages. Besides built-in multiline parsers, the latest version allows now to register our multiline parsers in the current parsers.conf files with the use of the new
Enhanced Splunk Output Connector
The new version sees the extension of the capabilities of our Splunk connector. With the update, it now supports additional metadata, fields and proper setup and raw mode behaviour.
New HTTP Health end point
We can now get more accurate information about how healthy Fluent Bit is using the new custom health endpoint
/api/v1/health. It now calculates error and success rates to indicate an excellent health status.
The new update shows that the Kubernetes filters use an internal cache to store Pods metadata for avoiding the generation of load in the API Server. With this version, we see a new option to enforce a cache TTL so the cached data can expire at specific intervals. Also, we can use the new option
kube_meta_cache_ttl to take advantage of this (disabled by default).
Fluentbit v1.8.1 is finally out now. Try it out here.