kafka 2.11-1.0.2 설치 그리고 오류

* 운영체제 : windows 10 64bit

* 카프카 버젼 :  kafka 2.11-1.0.2, kafka 2.11-2.0.0

첫째가 그려준 나야나

* zookeeper 시작
%KAFKA_HOME%/bin/windows/zookeeper-server-start.bat %KAFKA_HOME%/config/zookeeper.properties

* kafka 시작

%KAFKA_HOME%/bin/windows/kafka-server-start.bat %KAFKA_HOME%/config/server.properties

* /bin/windows 폴더로 이동
cd %KAFKA_HOME%\bin\windows

* 브로커 등록 테스트
zookeeper-shell.bat localhost:2181 ls /brokers/ids

* test topic 생성
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

* topic list 보기/조회
kafka-topics.bat --list --zookeeper localhost:2181


* console 프로듀서 시작
kafka-console-producer.bat --broker-list localhost:9092 --topic test

* console 컨슈머 시작
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning

* zookeeper 중지
%KAFKA_HOME%\bin\windows\zookeeper-server-stop.bat

설치하면서 발생한 error

프로듀서를 시작하고, 컨슈머를 시작한 후, 아래와 같은 에서 메세지가 뜨면서 프로듀서와 컨슈머 간에 메세지가 안갔다

host 문제 인줄 알고 댜양한 호스트(localhost,127.0.0.1,0.0.0.0,내IP)로 server.properties 를 변경하면서 실행해보다가 마무리 못하고 퇴근했다.   서버관리하는 남편님 덕분에 해결!

kafka 실행 창에 error 메세지가 떡하니 떠 있었음... 이.럴.수.가!

Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


* kafka 실행창에 아래와 같은 에러가 발생했다. java version 문제라고 한다.   java -d64 -version 으로 local을 확인해 보니 64bit 운영체제에 쌩뚱맞게 32bit java가 깔려 있었다.  지우고 java 64bit로 까니 잘 동작한다.  이걸로 왜 그리 헤매었나....기승전 에러메세지가 중요하다는 것을 또 한번 깨달음


[2023-03-10 00:22:48,918] ERROR [ReplicaManager broker=0] Error while making broker the leader for partition Topic: __consumer_offsets; Partition: 32; Leader: None; AllReplicas: ; InSyncReplicas:  in dir None (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while creating log for __consumer_offsets-32 in dir C:\tmp\kafka-logs3
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:944)
at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:68)
at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:53)
at kafka.log.LogSegment$.open(LogSegment.scala:634)
at kafka.log.Log.loadSegments(Log.scala:503)
at kafka.log.Log.<init>(Log.scala:237)
at kafka.log.Log$.apply(Log.scala:1981)
at kafka.log.LogManager$$anonfun$getOrCreateLog$1.apply(LogManager.scala:683)
at kafka.log.LogManager$$anonfun$getOrCreateLog$1.apply(LogManager.scala:651)
at scala.Option.getOrElse(Option.scala:121)
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:651)
at kafka.cluster.Partition$$anonfun$getOrCreateReplica$1.apply(Partition.scala:177)
at kafka.cluster.Partition$$anonfun$getOrCreateReplica$1.apply(Partition.scala:173)
at kafka.utils.Pool$$anon$2.apply(Pool.scala:61)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at kafka.utils.Pool.getAndMaybePut(Pool.scala:60)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:172)
at kafka.cluster.Partition$$anonfun$7$$anonfun$9.apply(Partition.scala:286)
at kafka.cluster.Partition$$anonfun$7$$anonfun$9.apply(Partition.scala:286)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.cluster.Partition$$anonfun$7.apply(Partition.scala:286)
at kafka.cluster.Partition$$anonfun$7.apply(Partition.scala:280)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:259)
at kafka.cluster.Partition.makeLeader(Partition.scala:280)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:1158)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:1156)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1156)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1076)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:185)
at kafka.server.KafkaApis.handle(KafkaApis.scala:110)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:941)

 

* java 삭제 방법
https://frogand.tistory.com/38

[Java] JDK 제거하는 법

 

* java.io.IOException: Map failed 에러 조치 방법, 제나나님 블로그 참고

https://jennana.tistory.com/179

[Error] Kafka server error : java.io IOException: Map failed 문제 해결하기

 

* 설치된 java 32bit/64bit 확인 방법, 코딩팩토리님 블로그 참고

https://coding-factory.tistory.com/732