What’s New in Apache Kafka Release 2.8?

Written by Harini Krish

Lead Technical Content Writer

Royal Cyber keeps up with the times and now supports the newly released Apache Kafka version 2.8, announced on April 19, 2021. Read on to find out what the new version means for you!

The 2.8.0 release includes many new features and improvements. This blog post highlights some of the major ones. Apache Kafka 2.8.0 is finally out and has early access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka relies on an internal Raft quorum that can be initiated through Kafka Raft metadata mode. The latest feature simplifies cluster administration and infrastructure management and marks a new era for Kafka itself.

Here is a glance at the most significant KIPs that have been merged:

  • KIP-500: lays out the vision for an event-driven model for managing metadata with a replicated log managed with the Raft protocol.

  • KIP-590: specifies a new protocol to allow forwarding client requests from brokers to the controller.

  • KIP-631: specifies the event-driven controller model, including the new broker lifecycle and the metadata record schemas.

  • KIP-595: specifies the Raft protocol, which is used for the @metadata topic.

What is New

KIP-696 Update Streams FSM to clarify ERROR state meaning
KIP-689 Extend StreamJoined to allow more store configs
KIP-680 TopologyTestDriver should not require a properties argument
KIP-671 Introduce Kafka Streams specific uncaught exception handler
KIP-663 API to start and shut down StreamThreads
KIP-572 Improve timeouts and retries in Kafka Streams
KIP-659 Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size
KIP-700 Add Describe Cluster API
KIP-684 Support mutual TLS authentication on SASL_SSL listeners
KIP-676 Respect logging hierarchy
KIP-673 Emit JSONs with new auto-generated schema
KIP-612 Limit broker connection creation rate
KIP-516 Topic identifiers
KIP-661 Expose task configurations in Connect REST API

What is KIP-500

Previously, Apache ZooKeeper was used by Kafka as a metadata store. Metadata for brokers and partitions were stored on the ZooKeeper quorum, which was in charge of the Kafka Controller election.

ZooKeeper is the external system to Kafka which means that it comes with its configuration arrangement, management tools, and best practices. So, if you wish to deploy a Kafka cluster, you will have to manage, deploy, and monitor Zookeeper. Since these two distributed systems need different configurations, the level of complexity rises, and as a result, it is much easier for system administrators to make errors. Having two systems also leads to duplication of work.

Having an external metadata store is unproductive in terms of resources as it needs to run additional processes. Moreover, this limits the scalability of Kafka itself. Each time the cluster is starting up, the Kafka controller must load the cluster from ZooKeeper.

The same goes on when a new controller is being elected. Given that the metadata gets more prominent over time, the loading of such metadata turns out to be more inefficient over time, heading to high loading processes. Thus, limiting the number of partitions the cluster can store up.

Introducing Kafka Raft Metadata Mode

Version 2.8.0 introduces early access to Zookeeper-Less Kafka as part of KPI-500. In the new release, ZooKeeper can be substituted by an internal Raft quorum of controllers. When Kafka Raft Metadata mode is enabled, Kafka will store its metadata and configurations into a topic called @metadata. This internal topic is handled by the internal quorum and reproduced across the cluster. The cluster nodes can now serve as brokers, controllers, or both (called combined nodes).

The removal of Apache ZooKeeper dependency makes the infrastructure management simpler for Kafka deployments. Kafka and ZooKeeper are two distinct services. Now that Kafka has been integrated so that it is not dependent on external services to serve as metadata stores, the learning curve will be shortened, which will help broaden Kafka’s adoption.

Moreover, this improvement provides a more scalable and robust overall architecture. As mentioned earlier, in ZooKeeper mode, Kafka had to store up to its metadata into ZooKeeper nodes. Each time the cluster was starting up or a controller election was happening, Kafka Controllers had to read the metadata from an ineffective external service. By substituting ZooKeeper with this internal Raft quorum, deployments can support more partitions.

Getting rid of the ZooKeeper dependency facilitates the support of clusters with a single node. When you want to test Kafka as part of a proof of concept, there is no need to start multiple processes.

Conclusion

The exclusion of Zookeeper dependency is a huge step forward for Kafka. The new KRaft mode feature will extend the scalability capabilities of Apache Kafka and reduce the learning curve as the teams do not have to worry about ZooKeeper. It will furthermore make Kafka configuration and deployment way more efficient and easier.

Apache Kafka 2.8.0 has a lot of amazing fixes and improvements in addition to the KIPs listed above. If you have not upgraded to the latest version of Apache Kafka, Sign up now for your free trial today! You can email us at info@royalcyber.com or visit us at www.royalcyber.com. In the meantime, make sure you follow us on our LinkedIn and Twitter accounts to stay up to date with our product and feature-related news.

Leave a Reply