Skip to main content
 首页 » 编程设计

apache-zookeeper之将 Kafka 0.8.1 与 Zookeeper 3.4.6 结合使用时遇到 LeaderNotAvailableException

2024年02月27日53yjmyzz

我按照他们的网站安装了 kafka 的稳定版本(0.8.1 和 2.9.2 Scala),并使用 3 节点 Zookeeper 整体(3.4.6)运行它。我尝试创建一个测试主题,但一直看到没有领导者分配给该主题的分区:

[kafka_2.9.2-0.8.1]$ ./bin/kafka-topics.sh --zookeeper <zookeeper_ensemble> --describe --topic test-1 
Topic:test-1    PartitionCount:1    ReplicationFactor:3 Configs: 
    Topic: test-1   Partition: 0    **Leader: none**    Replicas: 0,1,2 **Isr:**  

无论如何,我尝试使用控制台生成器写入该主题,但遇到了 LeaderNotAvailableException 异常:

[kafka_2.9.2-0.8.1]$ ./kafka-console-producer.sh --broker-list <broker_list> --topic test-1 
 
hello world 
 
[2014-04-22 11:58:48,297] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,321] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,322] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler) 
 
[2014-04-22 11:58:48,445] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,467] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,467] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler) 
 
[2014-04-22 11:58:48,590] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,612] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,612] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler) 
 
[2014-04-22 11:58:48,731] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,753] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,754] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler) 
 
[2014-04-22 11:58:48,876] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo) 
 
[2014-04-22 11:58:48,877] ERROR Failed to send requests for topics test-1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler) 
 
[2014-04-22 11:58:48,878] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) 
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. 
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) 
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) 
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) 
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) 
    at scala.collection.immutable.Stream.foreach(Stream.scala:547) 
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) 
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) 

我还应该指出,这最初工作了几天,然后突然创建的任何主题都出现了缺少领导者的问题。

请您参考如下方法:

Kafka 使用外部协调框架(默认为 Zookeeper)来维护配置。看来配置现在与 Kafka 日志数据不同步。在这种情况下,我将删除受影响的主题数据和相关的 Zookeeper 数据。

对于测试环境:

  1. 停止Kafka服务器Zookeeper服务器
  2. 移除两个服务的数据目录,默认为 /tmp/kafka-log/tmp/zookeeper
  3. 再次启动 Kafka-serverZookeeper-server
  4. 创建新主题

现在您可以再次处理该主题。

对于生产环境:

由于 Kafka 主题存储在不同的目录中,因此您应该删除特定的目录。您还应该使用 Zookeeper 客户端从 Zookeeper 中删除 /brokers/{broker_id}/topics/{broken_topic}

在做任何愚蠢的事情之前,请仔细阅读 Kafka 文档以确保配置结构。 Kafka正在推出删除主题功能( KAFKA-330 ),以便更轻松地解决问题。