Strmizi Kafka Cover

Strimzi : Master Apache Kafka on Kubernetes

by | 26.09.2021 | Engineering

There’s just an endless plethora of tools out there for the novice and the expert. Sometimes, it even seems like the open source community can’t catch a break until something new hits the web. This time, we’ll walk readers through another exciting addition to the Cloud Native Computing Foundation (CNCF) – Strimzi.

What Exactly is Strimzi

Originally inducted into the CNCF as a Sandbox Project back in 2019, Strimzi was created to help operators to manage the various components of another well known platform- Apache Kafka on Kubernetes. Readers may want to take a look back at our Kafka article to catch a better glimpse.

Strimzi became a standard for applications including Kafka brokers, Apache ZooKeeper, MirrorMaker and Kafka Connect. The end result was a better way to manage and control the Kafka platform and its native APIs, an issue that was commonly acknowledged by the community. One of these additions being the better management for topics and messages through HTTP, protocol supports, storage improvements and monitoring changes.

Strimzi Structure Overview
Strimzi Overview Source: Strimzi

How Strimzi Ties In With Kafka

Kafka, a creation of Linkedin dating back to early 2010s, was a message streaming tool for handling queuing systems and helping users manage large datasets on-time for intersecting with multiple social channels. Kafka runs on a similar structure for dataset handling, akin to SQL type of streams, rebranded as its own name- KSQL. A common problem often emerging out of Kafka’s usage has been its limitations for providing access control and distributing resources properly.

Most Kubernetes applications use a stream of operators to package, deploy and manage different elements in Kubernetes. Strimzi closes the gap on many of these resource needs by having the operators extend functionalities and automating tasks related to Kafka deployment. Users can thus apply Kafka changes and code applications through simplified CLI or GUI, without the complications of manually changing settings.

Strimzi Functionalities: What Makes It So Unique

Strimzi functions as a repository for container images, closely composed of Kubernetes elements. It also contains operators as previously discussed for getting Kafka to the Kubernetes platform. All operators designed for Kafka through Strimzi serve a specific purpose and are designed to resolve issues with control and deployment. Strimzi Operators are fundamental to the running of Strimzi.

Here is a small list of what the operators on Strimzi accomplish:-

  • Deploying and running Kafka clusters along with Kafka components through CLI commands as well as GUI interactions. Users can also now seamlessly connect Kafka clusters to other channels and perform changes on time, as opposed, to running with the default binary data formats that Kafka was notoriously designed to hold.
  • Configuring access to Kafka, its subsidiary channels and also all its individual components. This is most useful for controlling KSQL databases and performing changes as required by the operation or project.
  • Securing access to Kafka and helping users define protocols for specifying access control to repositories. A much needed introduction to the platform in the wake of multiple data breaches and vulnerabilities, Strimzi has a specialized platform for setting up keys and passwords for stream levels and assigns a hierarchy of safety checks based on user requirements.
  • Given the lack of large experts and documentations for running Kafka, which may have led most novices to discount it compared to its alternatives, upgrading Kafka hs become a hassle free task and gives users a simple list of commands to get latest changes on the server. What’s even better is that updates can now go undisturbed in the background without interfering with the Kafka clusters, a common problem reported by several users in the past.
  • Managing brokers and nodes that reside below and above the main stream edges. Having proper communication between the various elements of a Kafka network is important and Strimzi accomplishes that with a mix of interactive GUI and simple to use elements.
  • Strimzi also helps users to create topics and manage user interactions on social streams which required a much higher level of coding with previous configurations. Sometimes users would even have to log into different accounts to manage multiple topics, which is now all neatly wrapped in a single tabular framework.
Strimzi Overview Node Structure
Examples of node structures in Strimzi Source: Strimzi

A Treasure Trove of Operators: How To Best Use Strimzi

Knowing how to best use Strimzi requires knowing best how each of the operators can be applied for different needs and purposes:-

Cluster Operator

The cluster operator is closely related to operations for deploying and managing Apache Kafka clusters, Kafka Connect, Kafka MirrorMaker, Kafka Bridge, Kafka Exporter, and the Entity Operator.

The cluster operator is also, by its name, the central operator to help connect and control higher and lower level nodes in the Kafka network. This applies to the KSQL databases as well for configuring data join commands and deployment features to streams.

Entity Operator

Linked directly to the cluster operator and composed of the Topic Operator (manages Kafka Topics) and User Operator (manages Kafka Users), the entity operator can be regarded as an index level information for the databases, helping users know which streams are related to other streams.

It is also an important element for connecting multiple streams and broadcasting changes on time.

Topic Operator

The central operator for deploying and managing Kafka topics, which in the context of social channels and streams, refers to elements or information indexes that are linked to larger streams.

User Operator

Similar to the Topic Operator, this operator is linked to the former and helps users identify indexes for users linked to the topics that are related to the social streams. Users can easily update user requirements, change stream update frequencies, edit channel settings and so on.

It should be noted that the Cluster Operator can deploy the Topic Operator and User Operator as part of an Entity Operator configuration. What’s amazing is that all this can be achieved at the same time as a Kafka cluster.

Strimzi Opreator Overview
Strimzi Operators Source: Strimzi

A Much Tinier Issue: Where Some Work Is Still Needed

The fly in the ointment in the vast improvements that Strimzi brings to the table is the fact that it is still a native application platform that is very much open source. With a lack of commercial support and documentation for dealing with errors/issues, Strimzi requires a level of familiarity with the platform to master it perfectly.

While the community has made amazing strides in providing newcomers with the right tools and methods to best utilize Kafka and its streams, much is still left to be desired in terms of official releases, updates and compatibility issues.

Some criticism has also been lodged at the platform’s absolute dependence towards Kafka and its specific dataset structures which makes transfer to other formats difficult. Strimzi remedies this in some ways by using functionalities through the cluster operator but is still in extreme need of updates.

Final Notes and Review

All in all, Strimzi and Kafka seem to go well hand in hand, offering improvements for the much discussed platform. Strimzi as a cloud native application manages to hit the right notes in bringing the best features of Kafka, which were often left unexplored or unused altogether. Being a much younger name to the CNCF name however, does mean it has to go through the same process as its counterparts in the open source space to win over user engagement. Only time perhaps will tell if the community can bring its support and documentation to a more user friendly level.

Doing so would cut down the steep learning curve that Kafka already has. For more information on these two platforms, check out the official documentations and our articles, where we do a deeper analysis on similar tools or platforms. Feel free to contact us if you need help integrating Kafka with Strimzi.

Happy Learning!

CommunityNew

The DevOps Awareness Program

Subscribe to the newsletter

Join 100+ cloud native ethusiasts

#wearep3r

Join the community Slack

Discuss all things Kubernetes, DevOps and Cloud Native

Related articles6

How to clean up disk space occupied by Docker images?

How to clean up disk space occupied by Docker images?

Docker has revolutionised containers even if they weren't the first to walk the path of containerisation. The ease and agility docker provide makes it the preferred engine to explore for any beginner or enterprise looking towards containers. The one problem most of...

Parsing Packages with Porter

Parsing Packages with Porter

Porter works as a containerized tool that helps users to package the elements of any existing application or codebase along with client tools, configuration resources and deployment logic in a single bundle. This bundle can be further moved, exported, shared and distributed with just simple commands.

eBPF – The Next Frontier In Linux (Introduction)

eBPF – The Next Frontier In Linux (Introduction)

The three great giants of the operating system even today are well regarded as Linux, Windows and Mac OS. But when it comes to creating all purpose and open source applications, Linux still takes the reign as a crucial piece of a developer’s toolkit. However, you...

Falco: A Beginner’s Guide

Falco: A Beginner’s Guide

Falco shines through in resolving these issues by detecting and alerting any behaviour that makes Linux system calls. This system of alerting rules is made possible with the use of Sysdig’s filtering expressions to detect potentially suspicious activity. Users can also specify alerts for specific calls, arguments related to the calls and through the properties of the calling process.

Why DevOps Engineers Love Fluentd?

Why DevOps Engineers Love Fluentd?

Fluentd’s main operational forte lies in the exchange of communication and platforming for creating pipelines where log data can be easily transferred from log generators (such as a host or application) to their preferred destinations (data sinks such as Elasticsearch).

Operating On OpenTracing: A Beginner’s Guide

Operating On OpenTracing: A Beginner’s Guide

OpenTracing is a largely ignored variant of the more popular distributed tracing technique, commonly used in microservice architectures. Users may be familiar with the culture of using distributed tracing for profiling and monitoring applications. For the newcomers,...