Strmizi Kafka Cover

Strimzi : Master Apache Kafka on Kubernetes

by | 26.09.2021 | Engineering

There’s just an endless plethora of tools out there for the novice and the expert. Sometimes, it even seems like the open source community can’t catch a break until something new hits the web. This time, we’ll walk readers through another exciting addition to the Cloud Native Computing Foundation (CNCF) – Strimzi.

What Exactly is Strimzi

Originally inducted into the CNCF as a Sandbox Project back in 2019, Strimzi was created to help operators to manage the various components of another well known platform- Apache Kafka on Kubernetes. Readers may want to take a look back at our Kafka article to catch a better glimpse.

Strimzi became a standard for applications including Kafka brokers, Apache ZooKeeper, MirrorMaker and Kafka Connect. The end result was a better way to manage and control the Kafka platform and its native APIs, an issue that was commonly acknowledged by the community. One of these additions being the better management for topics and messages through HTTP, protocol supports, storage improvements and monitoring changes.

Strimzi Structure Overview
Strimzi Overview Source: Strimzi

How Strimzi Ties In With Kafka

Kafka, a creation of Linkedin dating back to early 2010s, was a message streaming tool for handling queuing systems and helping users manage large datasets on-time for intersecting with multiple social channels. Kafka runs on a similar structure for dataset handling, akin to SQL type of streams, rebranded as its own name- KSQL. A common problem often emerging out of Kafka’s usage has been its limitations for providing access control and distributing resources properly.

Most Kubernetes applications use a stream of operators to package, deploy and manage different elements in Kubernetes. Strimzi closes the gap on many of these resource needs by having the operators extend functionalities and automating tasks related to Kafka deployment. Users can thus apply Kafka changes and code applications through simplified CLI or GUI, without the complications of manually changing settings.

Strimzi Functionalities: What Makes It So Unique

Strimzi functions as a repository for container images, closely composed of Kubernetes elements. It also contains operators as previously discussed for getting Kafka to the Kubernetes platform. All operators designed for Kafka through Strimzi serve a specific purpose and are designed to resolve issues with control and deployment. Strimzi Operators are fundamental to the running of Strimzi.

Here is a small list of what the operators on Strimzi accomplish:-

  • Deploying and running Kafka clusters along with Kafka components through CLI commands as well as GUI interactions. Users can also now seamlessly connect Kafka clusters to other channels and perform changes on time, as opposed, to running with the default binary data formats that Kafka was notoriously designed to hold.
  • Configuring access to Kafka, its subsidiary channels and also all its individual components. This is most useful for controlling KSQL databases and performing changes as required by the operation or project.
  • Securing access to Kafka and helping users define protocols for specifying access control to repositories. A much needed introduction to the platform in the wake of multiple data breaches and vulnerabilities, Strimzi has a specialized platform for setting up keys and passwords for stream levels and assigns a hierarchy of safety checks based on user requirements.
  • Given the lack of large experts and documentations for running Kafka, which may have led most novices to discount it compared to its alternatives, upgrading Kafka hs become a hassle free task and gives users a simple list of commands to get latest changes on the server. What’s even better is that updates can now go undisturbed in the background without interfering with the Kafka clusters, a common problem reported by several users in the past.
  • Managing brokers and nodes that reside below and above the main stream edges. Having proper communication between the various elements of a Kafka network is important and Strimzi accomplishes that with a mix of interactive GUI and simple to use elements.
  • Strimzi also helps users to create topics and manage user interactions on social streams which required a much higher level of coding with previous configurations. Sometimes users would even have to log into different accounts to manage multiple topics, which is now all neatly wrapped in a single tabular framework.
Strimzi Overview Node Structure
Examples of node structures in Strimzi Source: Strimzi

A Treasure Trove of Operators: How To Best Use Strimzi

Knowing how to best use Strimzi requires knowing best how each of the operators can be applied for different needs and purposes:-

Cluster Operator

The cluster operator is closely related to operations for deploying and managing Apache Kafka clusters, Kafka Connect, Kafka MirrorMaker, Kafka Bridge, Kafka Exporter, and the Entity Operator.

The cluster operator is also, by its name, the central operator to help connect and control higher and lower level nodes in the Kafka network. This applies to the KSQL databases as well for configuring data join commands and deployment features to streams.

Entity Operator

Linked directly to the cluster operator and composed of the Topic Operator (manages Kafka Topics) and User Operator (manages Kafka Users), the entity operator can be regarded as an index level information for the databases, helping users know which streams are related to other streams.

It is also an important element for connecting multiple streams and broadcasting changes on time.

Topic Operator

The central operator for deploying and managing Kafka topics, which in the context of social channels and streams, refers to elements or information indexes that are linked to larger streams.

User Operator

Similar to the Topic Operator, this operator is linked to the former and helps users identify indexes for users linked to the topics that are related to the social streams. Users can easily update user requirements, change stream update frequencies, edit channel settings and so on.

It should be noted that the Cluster Operator can deploy the Topic Operator and User Operator as part of an Entity Operator configuration. What’s amazing is that all this can be achieved at the same time as a Kafka cluster.

Strimzi Opreator Overview
Strimzi Operators Source: Strimzi

A Much Tinier Issue: Where Some Work Is Still Needed

The fly in the ointment in the vast improvements that Strimzi brings to the table is the fact that it is still a native application platform that is very much open source. With a lack of commercial support and documentation for dealing with errors/issues, Strimzi requires a level of familiarity with the platform to master it perfectly.

While the community has made amazing strides in providing newcomers with the right tools and methods to best utilize Kafka and its streams, much is still left to be desired in terms of official releases, updates and compatibility issues.

Some criticism has also been lodged at the platform’s absolute dependence towards Kafka and its specific dataset structures which makes transfer to other formats difficult. Strimzi remedies this in some ways by using functionalities through the cluster operator but is still in extreme need of updates.

Final Notes and Review

All in all, Strimzi and Kafka seem to go well hand in hand, offering improvements for the much discussed platform. Strimzi as a cloud native application manages to hit the right notes in bringing the best features of Kafka, which were often left unexplored or unused altogether. Being a much younger name to the CNCF name however, does mean it has to go through the same process as its counterparts in the open source space to win over user engagement. Only time perhaps will tell if the community can bring its support and documentation to a more user friendly level.

Doing so would cut down the steep learning curve that Kafka already has. For more information on these two platforms, check out the official documentations and our articles, where we do a deeper analysis on similar tools or platforms. Feel free to contact us if you need help integrating Kafka with Strimzi.

Happy Learning!

CommunityNew

The DevOps Awareness Program

Subscribe to the newsletter

Join 100+ cloud native ethusiasts

#wearep3r

Join the community Slack

Discuss all things Kubernetes, DevOps and Cloud Native

Related articles6

Startup speed, enterprise quality

Startup speed, enterprise quality

Liebe Kunden, Partner und Kollegen,2021 ist vorbei und uns alle erwarten neue Herausforderungen und Ziele in 2022.In den letzten 3 Jahren hat sich p3r von einer One-Man-Show zu einer festen Größe im deutschen Cloud-Sektor entwickelt. Mit inzwischen 11...

Introduction to GitOps

Introduction to GitOps

GitOps serves to make the process of development and operations more developer-centric. It applies DevOps practices with Git as a single source of truth for infrastructure automation and deployment, hence the name “Git Ops.” But before getting deeper into what is...

Kaniko: How Users Can Make The Best Use of Docker

Kaniko: How Users Can Make The Best Use of Docker

Whether you love or hate containers, there are only a handful of ways to work with them properly that ensures proper application use with Docker. While there do exist a handful of solutions on the web and on the cloud to deal with all the needs that come with running...

Cilium: A Beginner’s Guide To Improve Security

Cilium: A Beginner’s Guide To Improve Security

A continuation from the previous series on eBPF and security concerns; it cannot be reiterated enough number of times how important it is for developers to ensure the safety and security of their applications. With the ever expanding reach of cloud and software...

How to clean up disk space occupied by Docker images?

How to clean up disk space occupied by Docker images?

Docker has revolutionised containers even if they weren't the first to walk the path of containerisation. The ease and agility docker provide makes it the preferred engine to explore for any beginner or enterprise looking towards containers. The one problem most of...

Parsing Packages with Porter

Parsing Packages with Porter

Porter works as a containerized tool that helps users to package the elements of any existing application or codebase along with client tools, configuration resources and deployment logic in a single bundle. This bundle can be further moved, exported, shared and distributed with just simple commands.