Kafka Broker Discovery and Zookeeper
Kafka Broker Discovery
- Every Kafka broker is called a bootstrap server.
- That means that you only need to connect to one broker and you will be connected to the entire cluster.
- Each broker knows about all brokers, topics and partitions (metadata)
- Zookeeper manages brokers (keeps a list of them)
- Zookeeper helps in performing leader election for partitions
- Zookeeper sends notifications to Kafka in case of changes (eg new topic, broker dies, broker comes up, delete topics etc)
- Kakfa can't work without Zookeeper
- Zookeeper by design operates with an odd number of servers(3, 5, 7)
- Zookeeper has a leader (handle writes) the rest of the servers are followers (handle reads)
- Zookeeper does NOT store consumer offsets with Kafka > v0.10)