Windows安装Kafka

下载与安装

下载地址：https://kafka.apache.org/downloads

配置

修改server.properties,指定Kafka日志存放目录

log.dirs=D:/Development/kafka/logs

# 端口设置
#listeners=PLAINTEXT://localhost:9092

若使用内置zookeeper则修改zookeeper.properties指定zookeeper数据存放目录与日志目录

dataDir=D:/Development/kafka/zkData
dataLogDir=D:/Development/kafka/zkLog

启动Zookeeper

可使用内置或外置Zookeeper，Windos建议使用内置zookeeper，简单方便

D:\Development\kafka\bin\windows>zookeeper-server-start.bat  ../../config/zookeeper.properties

启动Kafka

D:\Development\kafka\bin\windows>kafka-server-start.bat ../../config/server.properties

测试

创建Topic

D:\Development\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
Created topic mytopic.

生产者发送消息

D:\Development\kafka\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
>hello kafka
>

消费者接受消息

D:\Development\kafka\bin\windows>kafka-console-consumer.bat  --bootstrap-server localhost:9092 --from-beginning --topic mytopic
hello kafka

快速启动脚本

创建KafkaStart.bat文件，填入如下内容，在Window环境下快速启动Kafka

start cmd /k "D:\Development\kafka\bin\windows\zookeeper-server-start.bat D:\Development\kafka\config\zookeeper.properties"

start cmd /k "ping 127.1 -n "4">nul&&D:\Development\kafka\bin\windows\kafka-server-start.bat D:\Development\kafka\config\server.properties"

Linux安装单机版Kafka

下载与安装

下载地址：https://kafka.apache.org/downloads

wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

tar -xzf kafka_2.13-3.2.0.tgz

mv kafka_2.13-3.2.0 kafka

cd kafka

配置

:set number 或 :set nu开启行号显示

修改Kafka配置文件vim kafka/config/server.properties

24 broker.id=0
34 listeners=PLAINTEXT://localhost:9092
62 log.dirs=../logs
125 zookeeper.connect=localhost:2181

可使用外置或内置Zookeeper，这里使用内置Zookeeper

修改Zookeeper配置文件vim kafka/config/zookeeper.properties

16 dataDir=../zkData
17 dataLogDir=../zkLogs

添加环境变量

修改vim /etc/profile，添加Kafka环境变量设置

export KAFKA_HOME=/usr/local/program/kafka/
export PATH=$KAFKA_HOME/bin:$PATH

# 配置文件生效
source /etc/profile

启动内置zookeeper

这里启动内置Zookeeper，也可使用外置Zookeeper

zookeeper-server-start.sh config/zookeeper.properties

启动Kafka

# 前台运行
kafka-server-start.sh /usr/local/program/kafka/config/server.properties
# 后台运行
kafka-server-start.sh /usr/local/program/kafka/config/server.properties &

快速启动脚本

创建启动脚本：vim kafka_start.sh

# 启动zookeeper
zkServer.sh start
#等待5s,给与Zookeeper一定启动时间
sleep 5
#启动kafka
/usr/local/program/kafka/bin/kafka-server-start.sh /usr/local/program/kafka/config/server.properties
#等待5s后查看服务是否启动
sleep 5
jps

创建停止脚本：vim kafka_stop.sh

#关闭kafka
/usr/local/program/kafka/bin/kafka-server-stop.sh /usr/local/program/kafka/config/server.properties
#等待5s
sleep 3
#关闭zookeeper
zkServer.sh stop
#等待5s后查看服务是否关闭
sleep 5
jps

Kafka常用操作命令

新版Kafka相对于旧版Kafka操作命令有一定变化，表现在使用--bootstrap-server命令；

创建Topic

# 旧版
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic

# 新版
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic

查看队列分区明细

# 旧版
kafka-topics.sh --describe --zookeeper localhost:2181 --topic mytopic

# 新版
kafka-topics.sh --describe --bootstrap-server localhost:9092  --topic mytopic

查看队列列表

kafka-topics.sh --list --zookeeper localhost:2181

kafka-topics.sh --list --bootstrap-server localhost:9092 

删除toppic

kafka-topics.sh --delete --zookeeper localhost:2181  --topic mytopic2

kafka-topics.sh --delete --bootstrap-server localhost:9092  --topic mytopic2

修改分区数，只能往大改

kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --partitions 4

kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic mytopic --partitions 4

生产者发送消息

kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic

消费者消费消息

kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic mytopic 

kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic mytopic

Linux搭建Kafka集群

搭建Zookeeper集群

参考：https://blog.csdn.net/qq_38628046/article/details/123591295

下载与安装

下载地址：https://kafka.apache.org/downloads

wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

tar -xzf kafka_2.13-3.2.0.tgz

mv kafka_2.13-3.2.0 kafka

cd kafka

配置

24 broker.id=0
34 listeners=PLAINTEXT://node001:9092
62 log.dirs=../logs
125 zookeeper.connect=node001:2181,node002:2181,node003:2181

添加环境变量

修改vim /etc/profile，添加Kafka环境变量设置

export KAFKA_HOME=/usr/local/program/kafka/
export PATH=$KAFKA_HOME/bin:$PATH

# 配置文件生效
source /etc/profile

分发软件到其他节点

scp -r kafka root@node002:`pwd`
scp -r kafka root@node003:`pwd`

scp /etc/profile root@node002:/etc/profile
scp /etc/profile root@node003:/etc/profile

# node002、node003执行
source /etc/profile

修改配置

修改node002、node003机器上的server.properties配置文件，确保broker.id在每个节点机器上唯一。

node002

broker.id=1

node003

broker.id=2

启动集群

# 前台运行
kafka-server-start.sh /usr/local/program/kafka/config/server.properties
# 后台运行
kafka-server-start.sh /usr/local/program/kafka/config/server.properties &