一、kafka简介

详细介绍见： https://kafka.apache.org/0110/documentation.html

Apache Kafka® 是一个分布式流处理平台. 这到底意味着什么呢?
我们知道流处理平台有以下三种特性:

可以让你发布和订阅流式的记录。这一方面与消息队列或者企业消息系统类似。
可以储存流式的记录，并且有较好的容错性。
可以在流式记录产生时就进行处理。

Kafka适合什么样的场景?

它可以用于两大类别的应用:

构造实时流数据管道，它可以在系统或应用之间可靠地获取数据。 (相当于message queue)
构建实时流式应用程序，对这些流数据进行转换或者影响。 (就是流处理，通过kafka stream topic和topic之间内部进行变化)

为了理解Kafka是如何做到以上所说的功能，从下面开始，我们将深入探索Kafka的特性。.

首先是一些概念:

Kafka作为一个集群，运行在一台或者多台服务器上.
Kafka 通过 topic 对存储的流数据进行分类。
每条记录中包含一个key，一个value和一个timestamp（时间戳）。

Kafka有四个核心的API:

The Producer API 允许一个应用程序发布一串流式的数据到一个或者多个Kafka topic。
The Consumer API 允许一个应用程序订阅一个或多个 topic ，并且对发布给他们的流式数据进行处理。
The Streams API 允许一个应用程序作为一个流处理器，消费一个或者多个topic产生的输入流，然后生产一个输出流到一个或多个topic中去，在输入输出流中进行有效的转换。
The Connector API 允许构建并运行可重用的生产者或者消费者，将Kafka topics连接到已存在的应用程序或者数据系统。比如，连接到一个关系型数据库，捕捉表（table）的所有变更内容。

在Kafka中，客户端和服务器使用一个简单、高性能、支持多语言的 TCP 协议.此协议版本化并且向下兼容老版本，我们为Kafka提供了Java客户端，也支持许多其他语言的客户端。

在这里插入图片描述

二、kafka集群搭建

1、虚拟机准备

centos7镜像

node名	enp0s3	enp0s8	service
master	10.0.0.240	192.168.99.240	zk1,kafka1
worker1	10.0.0.240	192.168.99.240	zk2,kafka2
worker2	10.0.0.240	192.168.99.240	zk3,kafka3

2、docker安装

yum install -y wget
wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli

3、zookeeper集群搭建

docker启动命令如下

注意：为什么设置0.0.0.0，请参考https://stackoverflow.com/questions/30940981/zookeeper-error-cannot-open-channel-to-x-at-election-address

# zookeeper节点1
docker run -d --name zk1 -p 2181:2181 -p 2888:2888 -p 3888:3888 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-e ZOO_MY_ID=1 \
-e ZOO_SERVERS="server.1=0.0.0.0:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888" \
zookeeper:3.4.10

# zookeeper节点2
docker run -d --name zk2 -p 2181:2181 -p 2888:2888 -p 3888:3888 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-e ZOO_MY_ID=2 \
-e ZOO_SERVERS="server.1=zk1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zk3:2888:3888" \
zookeeper:3.4.10

# # zookeeper节点3
docker run -d --name zk3 -p 2181:2181 -p 2888:2888 -p 3888:3888 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-e ZOO_MY_ID=3 \
-e ZOO_SERVERS="server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=0.0.0.0:2888:3888" \
zookeeper:3.4.10

查看节点状态：

[root@master ~]# docker exec -it zk1 bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Mode: follower

[root@worker1 ~]# docker exec -it zk2 bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Mode: leader

[root@worker2 ~]# docker exec -it zk3 bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Mode: follower

4、kafka集群搭建

4.1、docker启动命令

由于位于不同主机，启动容器需要添加DNS解析，修改/etc/hosts文件，使用--add-host参数，如果不用DNS解析可以使用--net host使用宿主机网络，启动命令如下：

# kafka节点1
docker run -d --name kafka1 -p 9092:9092 \
-e KAFKA_BROKER_ID=1 \
-e KAFKA_ADVERTISED_HOST_NAME=broker1 \
-e KAFKA_ADVERTISED_PORT=9092 \
-e KAFKA_HOST_NAME=broker1 \
-e KAFKA_ZOOKEEPER_CONNECT=zk1:2181,zk2:2181,zk3:2181 \
-e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:9092" \
-e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://broker1:9092" \
-e KAFKA_DELETE_TOPIC_ENABLE=true \
--add-host broker1:10.0.0.240 \
--add-host broker2:10.0.0.241 \
--add-host broker3:10.0.0.242 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-v /var/run/docker.sock:/var/run/docker.sock \
wurstmeister/kafka:2.11-0.11.0.3

# kafka节点2
docker run -d --name kafka2 -p 9092:9092 \
-e KAFKA_BROKER_ID=2 \
-e KAFKA_ADVERTISED_HOST_NAME=broker2 \
-e KAFKA_ADVERTISED_PORT=9092 \
-e KAFKA_HOST_NAME=broker2 \
-e KAFKA_ZOOKEEPER_CONNECT=zk1:2181,zk2:2181,zk3:2181 \
-e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:9092" \
-e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://broker2:9092" \
-e KAFKA_DELETE_TOPIC_ENABLE=true \
--add-host broker1:10.0.0.240 \
--add-host broker2:10.0.0.241 \
--add-host broker3:10.0.0.242 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-v /var/run/docker.sock:/var/run/docker.sock \
wurstmeister/kafka:2.11-0.11.0.3

# kafka节点3
docker run -d --name kafka3 -p 9092:9092 \
-e KAFKA_BROKER_ID=3 \
-e KAFKA_ADVERTISED_HOST_NAME=broker3 \
-e KAFKA_ADVERTISED_PORT=9092 \
-e KAFKA_HOST_NAME=broker3 \
-e KAFKA_ZOOKEEPER_CONNECT=zk1:2181,zk2:2181,zk3:2181 \
-e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:9092" \
-e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://broker3:9092" \
-e KAFKA_DELETE_TOPIC_ENABLE=true \
--add-host broker1:10.0.0.240 \
--add-host broker2:10.0.0.241 \
--add-host broker3:10.0.0.242 \
--add-host zk1:10.0.0.240 \
--add-host zk2:10.0.0.241 \
--add-host zk3:10.0.0.242 \
-v /var/run/docker.sock:/var/run/docker.sock \
wurstmeister/kafka:2.11-0.11.0.3

4.2、查看是否注册到zookeeper

查看zookeeper中brokers的id数是否为[1, 2, 3]

[root@master ~]# docker exec -it zk1 bash
bash-4.4# bin/zkCli.sh 
Connecting to localhost:2181
2020-08-11 06:37:04,545 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2020-08-11 06:37:04,549 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=ef834bfc860a
2020-08-11 06:37:04,549 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
2020-08-11 06:37:04,551 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/zookeeper-3.4.13/bin/../build/classes:/zookeeper-3.4.13/bin/../build/lib/*.jar:/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/conf:
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-1127.10.1.el7.x86_64
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2020-08-11 06:37:04,552 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2020-08-11 06:37:04,553 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/zookeeper-3.4.13
2020-08-11 06:37:04,553 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2020-08-11 06:37:04,577 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2020-08-11 06:37:04,665 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2020-08-11 06:37:04,676 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1303] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100000e53150004, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]
[zk: localhost:2181(CONNECTED) 1] ls /brokers/ids
[1, 2, 3]

三、docker-compose快速部署

version: '3'
services:
  zk1:
    image: zookeeper:3.4.13
    restart: always
    container_name: zk1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888 server.4=zk4:2888:3888:observer

  zk2:
    image: zookeeper:3.4.13
    restart: always
    container_name: zk2
    ports:
      - "2182:2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888 server.4=zk4:2888:3888:observer

  zk3:
    image: zookeeper:3.4.13
    restart: always
    container_name: zk3
    ports:
      - "2183:2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888 server.4=zk4:2888:3888:observer

  zk4:
    image: zookeeper:3.4.13
    restart: always
    container_name: zk4
    ports:
      - "2184:2181"
    environment:
      ZOO_MY_ID: 4
      PEER_TYPE: observer
      ZOO_SERVERS: server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888 server.4=zk4:2888:3888:observer

  zkweb:
    image: peterpoker/zkweb:v1.2.1
    restart: always
    container_name: zkweb
    ports:
      - "8099:8099"
                        
  broker1:
    image: wurstmeister/kafka:2.12-2.4.1
    restart: always
    container_name: broker1
    ports:
      - "9091:9092"
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: broker1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_HOST_NAME: broker1
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181,zk4:2181
      KAFKA_LISTENERS: PLAINTEXT://broker1:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

  broker2:
    image: wurstmeister/kafka:2.12-2.4.1
    restart: always
    container_name: broker2
    ports:
      - "9092:9092"
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_HOST_NAME: broker2
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_HOST_NAME: broker2
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181,zk4:2181
      KAFKA_LISTENERS: PLAINTEXT://broker2:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

  broker3:
    image: wurstmeister/kafka:2.12-2.4.1
    restart: always
    container_name: broker3
    ports:
      - "9093:9092"
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_HOST_NAME: broker3
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_HOST_NAME: broker3
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181,zk4:2181
      KAFKA_LISTENERS: PLAINTEXT://broker3:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

  kafka-manager:
    image: sheepkiller/kafka-manager:alpine
    restart: always
    container_name: kafka-manager
    ports:
      - "9000:9000"
    depends_on:
      - broker1
      - broker2
      - broker3
    environment:
      ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181,zk4:2181

原文链接：https://blog.csdn.net/cyfblog/article/details/107755957