Kafka Broker Discovery and Zookeeper

Kafka Broker Discovery

  • Every Kafka broker is called a bootstrap server.
  • That means that you only need to connect to one broker and you will be connected to the entire cluster.
  • Each broker knows about all brokers, topics and partitions (metadata)

kafka broker_discovery

Zookeeper

  • Zookeeper manages brokers (keeps a list of them)
  • Zookeeper helps in performing leader election for partitions
  • Zookeeper sends notifications to Kafka in case of changes (eg new topic, broker dies, broker comes up, delete topics etc)
  • Kakfa can't work without Zookeeper
  • Zookeeper by design operates with an odd number of servers(3, 5, 7)
  • Zookeeper has a leader (handle writes) the rest of the servers are followers (handle reads)
  • Zookeeper does NOT store consumer offsets with Kafka > v0.10)

kafka zookeeper