kafka吞吐量测试

1、测试生产者的吞吐量

[root@Hexindai-C11-71 ~]#  kafka-producer-perf-test.sh --num-records 10000000 --record-size 1000 --topic test2 --throughput 100000  --producer-props bootstrap.servers=172.20.11.71:9092,172.20.11.72:9092,172.20.11.73:9092

相关参数说明：
—topic
指定Kafka集群的topic名称，本例为test2
–num-records
　　总共需要发送的消息数，本例为10000000
–record-size
是一条信息有多大，单位是字节
–throughput
每秒钟发送的记录数，本例为100000
–producer-props bootstrap.servers=172.20.11.71:9092,172.20.11.72:9092,172.20.11.73:9092
很显然是kafka集群的broker地址

生产压力测试过程如下：

352129 records sent, 70425.8 records/sec (67.16 MB/sec), 395.4 ms avg latency, 464.0 max latency.
344976 records sent, 68995.2 records/sec (65.80 MB/sec), 457.2 ms avg latency, 778.0 max latency.
371616 records sent, 74323.2 records/sec (70.88 MB/sec), 447.1 ms avg latency, 821.0 max latency.
370032 records sent, 74006.4 records/sec (70.58 MB/sec), 450.5 ms avg latency, 723.0 max latency.
422864 records sent, 84572.8 records/sec (80.65 MB/sec), 367.7 ms avg latency, 648.0 max latency.
349632 records sent, 69926.4 records/sec (66.69 MB/sec), 492.5 ms avg latency, 715.0 max latency.
428672 records sent, 85734.4 records/sec (81.76 MB/sec), 382.3 ms avg latency, 458.0 max latency.
410976 records sent, 82195.2 records/sec (78.39 MB/sec), 396.8 ms avg latency, 567.0 max latency.
398288 records sent, 79657.6 records/sec (75.97 MB/sec), 413.1 ms avg latency, 657.0 max latency.
451552 records sent, 90310.4 records/sec (86.13 MB/sec), 363.3 ms avg latency, 398.0 max latency.
413376 records sent, 82675.2 records/sec (78.85 MB/sec), 395.6 ms avg latency, 653.0 max latency.
459712 records sent, 91942.4 records/sec (87.68 MB/sec), 356.4 ms avg latency, 374.0 max latency.
433840 records sent, 86768.0 records/sec (82.75 MB/sec), 377.1 ms avg latency, 436.0 max latency.
407184 records sent, 81436.8 records/sec (77.66 MB/sec), 401.3 ms avg latency, 741.0 max latency.
443808 records sent, 88761.6 records/sec (84.65 MB/sec), 371.3 ms avg latency, 445.0 max latency.
390016 records sent, 78003.2 records/sec (74.39 MB/sec), 419.9 ms avg latency, 694.0 max latency.
458080 records sent, 91616.0 records/sec (87.37 MB/sec), 357.1 ms avg latency, 390.0 max latency.
424032 records sent, 84485.4 records/sec (80.57 MB/sec), 377.9 ms avg latency, 508.0 max latency.
392224 records sent, 78444.8 records/sec (74.81 MB/sec), 426.2 ms avg latency, 686.0 max latency.
453808 records sent, 90761.6 records/sec (86.56 MB/sec), 363.2 ms avg latency, 426.0 max latency.
391424 records sent, 78284.8 records/sec (74.66 MB/sec), 418.7 ms avg latency, 674.0 max latency.
450912 records sent, 90182.4 records/sec (86.00 MB/sec), 362.6 ms avg latency, 386.0 max latency.
426416 records sent, 85283.2 records/sec (81.33 MB/sec), 385.0 ms avg latency, 434.0 max latency.
365200 records sent, 73040.0 records/sec (69.66 MB/sec), 448.3 ms avg latency, 1126.0 max latency.
10000000 records sent, 81878.623129 records/sec (78.09 MB/sec), 397.61 ms avg latency, 1126.00 ms max latency, 372 ms 50th, 617 ms 95th, 740 ms 99th, 1121 ms 99.9th.

参数解析：

本例中一共写入1千万条消息，平均是81878.623129条消息/秒，每秒向Kafka写入了78.09MB的数据，每次写入的平均延迟为397.61.毫秒，最大的延迟为1126.00毫秒。
2、测试kafka消费的吞吐量

[root@Hexindai-C11-71 ~]#  kafka-consumer-perf-test.sh --broker-list 172.20.11.71:9092  --topic kafka01 --messages 1000000 --fetch-size 1048576 --threads 10
[root@Hexindai-C11-71 ~]#

相关参数说明：
–topic
指定topic的名称，指定消费的topic,本例为 test2

–fetch-size

指定每次fetch的数据的大小，本例为1048576，也就是1M

–messages 　　　　
　　总共要消费的消息个数，本例为1000000，100w

–threads
　　指定消费的线程数为10

测试结果说明：

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2021-11-05 16:38:10:312, 2021-11-05 16:38:26:303, 954.0787, 59.6635, 1000424, 62561.6910, 3136, 12855, 74.2185, 77823.7262