kafka 3.5.0 raft協議安裝

前言

最近做項目，需要使用kafka進行通信，且只能使用kafka，筆者沒有測試集群，就自己搭建了kafka集群，實際上筆者在很早之前就搭建了，因為當時還是zookeeper（簡稱ZK）注冊元數據，現在新版kafka（3.0.0開始）已經自帶了元數據能力（使用raft協議）減少了kafka對zk的依賴性。筆者在查詢資料發現，說jdk至少jdk11，實測jdk8也能運行，且并不需要網上說的3+4節點，3+3即可，當然理論上broker節點越多越好，但是元數據節點建議3、5個最合適，raft的過半一致性和容錯性的綜合取舍。

準備

準備kafka安裝包：Apache Kafka

筆者使用的kafka 3.5.0和scala 2.13，采用3臺虛擬機，當然容器也不是不行，注意持久化pv pvc和配置的管理（ip換成域名，dns的切換支持），中間件建議使用虛擬機，可以降低很多容錯性。

jdk使用open jdk，配置java_home和path，以Ubuntu為例

?sudo apt install openjdk-8-jdk-headless

以macOS為例，創建一個ubuntu-server 最小安裝的虛擬機（vmware，畢竟個人使用不要錢），然后安裝openssh 和 openjdk，然后shutdown now

網絡選擇橋接，相當于一臺“真實在”網絡上的一臺物理機

這樣就得到了

192.168.0.108

192.168.0.107

192.168.0.106

3臺虛擬機

步驟

先看kafka集群的架構圖，實際上安裝的過程就是架構圖的執行過程

從圖中可以看出已經沒有zk的存在了，從kafka節點自己管理元數據，通過raft協議選主的方式。

1. kafka的準備

上傳kafka安裝包，必須是二進制安裝包，不要源碼包，編譯比較麻煩，然后解壓

tar -zxvf ?kafka_2.13-3.5.0.gz

查看配置目錄會發現

明顯多了kraft的配置目錄，那么如果使用kafka raft元數據中心，則需要修改kraft目錄，啟動時指定kraft目錄的配置

2. 配置修改

raft協議實際上跟zk差不多，使用raft協議的中間件就太多了，但是本質上每個節點都需要一個唯一id，zk也是如此，所以kafka kraft就相當于集成的zk。

在kraft下的有3個文件文件，其中啟動相關的是server.properties中

執行配置修改


# The role of this server. Setting this puts us in KRaft mode
process.roles=broker,controller# The node id associated with this instance's roles
node.id=1# The connect string for the controller quorum
controller.quorum.voters=1@localhost:9093############################# Socket Server Settings ############################## The address the socket server listens on.
# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092,CONTROLLER://:9093# Name of listener used for communication between brokers.
inter.broker.listener.name=PLAINTEXT# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://localhost:9092# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.names=CONTROLLER# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600############################# Log Basics ############################## A comma separated list of directories under which to store log files
log.dirs=/tmp/kraft-combined-logs# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1############################# Log Flush Policy ############################## Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000############################# Log Retention Policy ############################## The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

每一行都有注釋，重點關注

筆者設定

192.168.0.106 nodeid 1?

192.168.0.107 nodeid 2

192.168.0.108 nodeid 3

至此配置基本上完成，同理一個節點可以同時是controller和broker，也可以僅僅是controller或者broker，因為controller的負載比較輕，所以一般是和broker一起。其中有個log.dir這個的路徑是下面元數據生成的路徑（選主）和數據事務日志，索引日志的存儲目錄

3. 啟動

1. 生成uuid

任意找一個節點執行：

./kafka-storage.sh random-uuid

每次執行uuid會不一樣，這個uuid標識是一個集群，所以所有節點公用一個uuid，不要每個節點重新生成，會識別不了?

然后執行format，如下標紅是我生成的，這個每次不是固定的

?./kafka-storage.sh format -t gZzkfRm4T1y8wSAY-ZNG5Q -c ../config/kraft/server.properties??

?格式化配置文件，同步其他節點

配置文件有什么變化？在日志配置的目錄下出現

關鍵還是meta的文件，有集群id和節點id，版本號，這個對啟動至關重要。

即在上面的log.dir的目錄生成，所以盡量不能使用臨時目錄

2. 啟動

啟動就很簡單了，使用剛剛配置的server.properties執行啟動即可

./kafka-server-start.sh -daemon ../config/kraft/server.properties

不過為了方便查看啟動日志，建議執行日志的console文件輸出

?先看事務日志和索引

驗證

驗證很簡單，查看bin同級目錄下的日志即可

日志帶有[2025-02-08 08:34:12,286] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)?

如果生成用途可以安裝kafka的控制臺，kafka-ui，不過我這里就不安裝了，因為docker安裝比較容易。

總結

kafka從3.0.0開始推出了raft模式的元數據中心，實際上類似zk，kafka自己命名kraft。使用這種方式搭建kafka集群將不再需要zk，同理，kafka的集群的每個節點可以同時是broker和controller（以前zk充當），也可以是單獨的broker，controller（負載不重，不建議單獨controller，跟zk沒區別），官方說明需要jdk11及以上，實測jdk8可以運行，但是生成建議嚴格按照官方標定的jdk執行，jdk是向下兼容的，但是不確定是否會涉及新api或新特性的使用。

另外實際使用中，可能會涉及使用iptables做nat限制kafka的連接方，比如在kafka節點通過iptables限制發送者或者消費端的ip

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 9093 -j DNAT --to-destination kafkaxxx:9093

kafkaxxx --- 指定的是 Kafka 服務所在的機器地址

如果kafka是對接方提供，則在nat打通時，需要客戶端連接的服務器也執行iptables，否則可能出現連接kafka正常，但是不能消費。

iptables -t nat -A POSTROUTING -p tcp -m tcp --dport 9093 -j SNAT --to-source natxxx

natxxx ---?指定的是配置 iptables 的本機的地址