我按照他们的网站安装了 kafka 的稳定版本(0.8.1 和 2.9.2 Scala),并使用 3 节点 Zookeeper 整体(3.4.6)运行它。我尝试创建一个测试主题,但一直看到没有领导者分配给该主题的分区:
[kafka_2.9.2-0.8.1]$ ./bin/kafka-topics.sh --zookeeper <zookeeper_ensemble> --describe --topic test-1
Topic:test-1 PartitionCount:1 ReplicationFactor:3 Configs:
Topic: test-1 Partition: 0 **Leader: none** Replicas: 0,1,2 **Isr:**
无论如何,我尝试使用控制台生成器写入该主题,但遇到了 LeaderNotAvailableException 异常:
[kafka_2.9.2-0.8.1]$ ./kafka-console-producer.sh --broker-list <broker_list> --topic test-1
hello world
[2014-04-22 11:58:48,297] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,321] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,322] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
[2014-04-22 11:58:48,445] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,467] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,467] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
[2014-04-22 11:58:48,590] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,612] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,612] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
[2014-04-22 11:58:48,731] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,753] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,754] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
[2014-04-22 11:58:48,876] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-04-22 11:58:48,877] ERROR Failed to send requests for topics test-1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-04-22 11:58:48,878] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:547)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
我还应该指出,这最初工作了几天,然后突然创建的任何主题都出现了缺少领导者的问题。
请您参考如下方法:
Kafka 使用外部协调框架(默认为 Zookeeper)来维护配置。看来配置现在与 Kafka 日志数据不同步。在这种情况下,我将删除受影响的主题数据和相关的 Zookeeper 数据。
对于测试环境:
- 停止
Kafka服务器
和Zookeeper服务器
- 移除两个服务的数据目录,默认为
/tmp/kafka-log
和/tmp/zookeeper
。 - 再次启动
Kafka-server
和Zookeeper-server
- 创建新主题
现在您可以再次处理该主题。
对于生产环境:
由于 Kafka 主题存储在不同的目录中,因此您应该删除特定的目录。您还应该使用 Zookeeper 客户端从 Zookeeper 中删除 /brokers/{broker_id}/topics/{broken_topic}
。
在做任何愚蠢的事情之前,请仔细阅读 Kafka 文档以确保配置结构。 Kafka正在推出删除主题功能( KAFKA-330 ),以便更轻松地解决问题。