Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale

by Gwen Shapira, Todd Palino, Rajini Sivaram

Every enterprise application creates data, whether it consists of log messages, metrics, user activity, or outgoing messages. Moving all this data is just as important as the data itself. With this updated edition, application architects, developers, and production engineers new to the Kafka streaming platform will learn how to handle data in motion. Additional chapters cover Kafka's AdminClient API, transactions, new security features, and tooling changes.

Engineers from Confluent and LinkedIn responsible for developing Kafka explain how to deploy production Kafka clusters, write reliable event-driven microservices, and build scalable stream processing applications with this platform. Through detailed examples, you'll learn Kafka's design principles, reliability guarantees, key APIs, and architecture details, including the replication protocol, the controller, and the storage layer.

You'll

Best practices for deploying and configuring KafkaKafka producers and consumers for writing and reading messagesPatterns and use-case requirements to ensure reliable data deliveryBest practices for building data pipelines and applications with KafkaHow to perform monitoring, tuning, and maintenance tasks with Kafka in productionThe most critical metrics among Kafka's operational measurementsKafka's delivery capabilities for stream processing systems

Genres: Computer Science, Programming, Technology, Nonfiction, Software, Engineering, Technical

457 pages, Kindle Edition

Published November 5, 2021

Reviews

[alper · Rating 5 out of 5]

Adeta bir şaheser. Kafka ile bir şekilde münasebeti olan herkes muhakkak okumalı. Özellikle "Kafka Internals" başlığı. Ayrıca "Distributed system" üzerine de epey iç görü sağlar."-ilities" kardeşler başta olmak üzere. :) Repertuara yeni çözümler eklemek her zaman sevindirici.

Bir de eleştirim olsun. Kafka "source of truth" değil, bence olmamalı. Fazla kasmışlar o taraflara, gereksiz olmuş... Ha bir de nasıl config yaparsan yap, network gerçeği var. "Strong consistency" senaryosunda O outbox yapılacak. :P

Sıfırdan başlayacak olanlara da spring / spring kafka doc. bence çok güzel bir kaynak. Oradan sonra buna geçilebilir. Onun kaynak kodları, testleri de adeta derstir...

Ben şimdi bir süre kafka kaynak kodlarını inceleyeceğim, monitoring vs.. Epey ek iş çıktı kitaptan. :)

[Matt Chequers · Rating 4 out of 5]

Didn't read thoroughly, as I was more interested in high level overview of Kafka and don't have an immediate use case. Great overview and ability to drill down further with hands on examples if one wishes.

[Akhilesh Kumar · Rating 5 out of 5]

Read the chapters selectively and not the entire book thoroughly.
For someone who has worked on Kafka but is more interested in going through the internals and the design philosophy of Kafka, I recommend a few chapters as must read:

1. Ch 1 - Meet Kafka (almost a refresher on a high level)
2. Ch 2 - Especially the config and hardware selection part. A refresher on how resources should be given for scaling the software for those who are already familiar. For the beginners, it is suggested to read and understand the calculations for resources and computations.
3. Ch 3 / Ch 4 - Producers and Consumers
4. Ch 5 - I just skimmed through it because it's mostly about the administration.
5. Ch 6 / Ch 7 / Ch 8 - These chapters are on the dive deeps on the internals of Kafka. Loved these absolutely.
6. Ch 9, 10, 11, 12 - Skimmed through them
7. Ch 13 - Monitoring metrics for Kafka, interesting one.
8. Ch 14 - Very insightful on Kafka stream processing.

Book is pretty straightforward, and definitely justifies the title of being a definitive guide. 💯

[Miloš · Rating 5 out of 5]

Truly a definitive guide

Be it the overall architecture, client protocols or delivery guarantees, this book manages to strike a perfect balance between readability and the level of detail to go into; an additional bonus is that authors had production in mind at all times, and especially in the monitoring chapter. The streams chapter is understandably just a tease since books have been written on the subject (and many of them referenced).

I have no doubt I'll be returning to it.

[Michal Sokolowski · Rating 4 out of 5]

Over all the book is quite informative, even more so when you are a Java dev who can use code snippets directly. However, if you are primarily using language other than Java the book contains a lot of pages with little to no value to you. As a Golang dev, I found it very irritating.

I wish the author had separated theory from practice (in Java), the book would far easier to read.

[Senjin Hajrulahovic · Rating 5 out of 5]

Great book. Covers a lot of topics and offers a broad overview of the kafka ecosystem. It's worth reading even if you read the first edition, since since then a lot of things changed. Decreasing the dependency on apache zookeeper being one of them. The chapter on monitoring is really valuable.

[Nickolai · Rating 3 out of 5]

Очень много теории, которая порой повторяется, а порой уходит в такие дремучие дебри, что будет полезна только тем, кто устанавливает и настраивает кафку. Для простого пользователя книга слишком сложна. К тому же, не всегда достаточно приводимых примеров использования.

[Prem · Rating 5 out of 5]

Loaded with details and presented in interesting style

[Vitali · Rating 3 out of 5]

mostly a reference. little architecture patterns and how to use in real life.
serves not very good as intro