k8s部署rabbitMQ集群

一. 使用statefulset部署rabbitmq集群

在部署集群之前需要为集群创建一个Storage Class（存储类）来作为集群数据的持久化后端。本例中使用ceph作为后端存储，在创建存储类之前需要先搭建好，并保证在k8s集群各个节点上均能挂载该ceph存储。

1.创建命名空间rabbit-mq


# cat rabbit-mq.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: rabbit-mq


# kubectl apply -f rabbit-mq.yaml

2. Service Account是Kubernetes Pod中的程序用于访问Kubernetes API的Account(账号)，它为Pod中的程序提供访问Kubernetes API的身份标识。下面我们创建rabbitmq Pod的ServiceAccount，并针对Kubernetes的endpoint资源做授权，创建相关的role和rolebinding。

在Kubernetes上创建rabbitmq这个ServiceAccount以及相关的role和rolebinding.创建如下的rabbitmq.rbac.yaml文件

# cat rabbitmq.rbac.yaml
---
apiVersion: v1
kind: ServiceAccount  #集群访问apiserver的凭证
metadata:
  name: rabbitmq
  namespace: rabbit-mq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: endpoint-reader
  namespace: rabbit-mq
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding  #将角色绑定
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: endpoint-reader
  namespace: rabbit-mq
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

# kubectl apply -f rabbitmq.rbac.yaml
[root@master rabbitmq]# kubectl get ServiceAccount -n rabbit-mq
NAME       SECRETS   AGE
default    1         60m
rabbitmq   1         60m

3. 创建访问rabbitmq集群的service，创建rabbitmq.service.yaml

# cat rabbitmq.service.yaml
---
kind: Service
apiVersion: v1
metadata:
  name: rabbitmq-headless
  namespace: rabbit-mq
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
   - name: amqp
     port: 5672
   - name: http
     port: 15672
  selector:
    app: rabbitmq
    
---
kind: Service
apiVersion: v1
metadata:
  namespace: rabbit-mq
  name: rabbitmq-service
spec:
  ports:
  - name: http
    protocol: TCP
    port: 15672
    nodePort: 35672  #管理web界面
  - name: amqp
    protocol: TCP
    port: 5672
    targetPort: 5672
    nodePort: 35671
  selector:
    app: rabbitmq
  type: NodePort


# kubectl create -f rabbitmq.service.yaml
[root@master rabbitmq]# kubectl get svc -n rabbit-mq
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
rabbitmq-headless   ClusterIP   None             <none>        5672/TCP,15672/TCP               58m
rabbitmq-service    NodePort    10.111.143.240   <none>        15672:35672/TCP,5672:35671/TCP   58m

4.创建一个secret对象，用来存储rabbitmq的用户名、密码及erlang.cookie。

erlang创建步骤：erlang是集群之间互访的秘钥
# 首先需要生成一个erlang.cookie的文件： echo $(openssl rand -base64 32) > erlang.cookie
# 然后再生成base64的值如：echo -n 'v/sWCz4uKETUvneRyJVn87Jg15si2eGaWg54Yvefhrk=' |base64
# 最后写一个secret对象的yaml。 devsecret.yaml

# cat devsecret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: devsecret
  namespace: rabbit-mq
type: Opaque
data:
  rabbitDefaulUser: "cm9vdA=="  # echo -n 'root' |base64
  rabbitDefaultPass: "Snl5MTIzNDUhQCM="
  erlang.cookie: "di9zV0N6NHVLRVRVdm5lUnlKVm44N0pnMTVzaTJlR2FXZzU0WXZlZmhyaz0="


# kubectl apply -f devsecret.yaml
[root@master rabbitmq]# kubectl get secrets -n rabbit-mq
NAME                   TYPE                                  DATA   AGE
devsecret              Opaque                                3      59m

5.Configmap创建 rabbitmq配置文件

# cat rabbitmq-config.yaml
---
#Configmap创建 rabbitmq配置文件
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: rabbit-mq
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
#启用插件rabbitmq_management和rabbitmq_peer_discovery_k8s
  rabbitmq.conf: |
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      #################################################
      # rabbit-mq is rabbitmq-cluster's namespace#
      #################################################
      cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbit-mq.svc.cluster.local
      cluster_formation.node_cleanup.interval = 30
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      queue_master_locator=min-masters
      cluster_formation.randomized_startup_delay_range.max = 2
      vm_memory_high_watermark.absolute = 1GB
      disk_free_limit.absolute = 2GB
      loopback_users.guest = false

# kubectl apply -f rabbitmq-config.yaml
[root@master rabbitmq]# kubectl get cm -n rabbit-mq
NAME              DATA   AGE
rabbitmq-config   2      63m

6.通过statefulset类型创建rabbitmq集群，创建rabbitmq.statefulset.yaml

# cat rabbitmq-headless.yaml
--- 
#statefulset创建（podAntiAffinity反亲和性）
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbit-mq
spec:
  serviceName: rabbitmq-headless
  selector:
    matchLabels:
      app: rabbitmq  #在apps/v1中，需与 .spec.template.metadata.label 相同，用于hostname传播访问pod
  replicas: 3  #副本数3
  template:
    metadata:
      labels:
        app: rabbitmq
      annotations:
        scheduler.alpha.kubernetes.io/affinity: >
            {
              "podAntiAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": [{
                  "labelSelector": {
                    "matchExpressions": [{
                      "key": "app",
                      "operator": "In",
                      "values": ["rabbitmq"]
                    }]
                  },
                  "topologyKey": "kubernetes.io/hostname"
                }]
              }
            }
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:
      - name: rabbitmq
        image: jinyuyun.top/rabbitmq:management
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 1
            memory: 1Gi
        volumeMounts:
          - name: config-volume
            mountPath: /etc/rabbitmq
          - name: rabbitmq-data
            mountPath: /var/lib/rabbitmq/mnesia
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 5
        imagePullPolicy: Always
        env:
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              key: rabbitDefaulUser
              name: devsecret                     #登陆用户名和密码都存储在一个secret对象中
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              key: rabbitDefaultPass
              name: devsecret
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: devsecret
              key: erlang.cookie
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: K8S_SERVICE_NAME
          value: "rabbitmq-headless"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(HOSTNAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
        - name: K8S_HOSTNAME_SUFFIX
          value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
        #- name: RABBITMQ_ERLANG_COOKIE
          #value: "unique-erlang-cookie"
          #value: "fOfDsEo6sa+n50EJZ0nAjP7342Y4gSk1gBVvyArrzUY="  # generator by: echo $(openssl rand -base64 32)
      volumes:
      - name: config-volume
        configMap:
          name: rabbitmq-config
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
    spec:
      storageClassName: "jyy-cephfs-sc"
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 2Gi

# kubectl apply -f rabbitmq-headless.yaml
[root@master rabbitmq]# kubectl get pod -n rabbit-mq
NAME         READY   STATUS    RESTARTS   AGE
rabbitmq-0   1/1     Running   0          57m
rabbitmq-1   1/1     Running   0          55m
rabbitmq-2   1/1     Running   0          54m

二. 集群自动加入.此过程时间有点长,直到状态变为running即为加入集群成功

我们可以进任意一个pod日志里面可以看到，RabbitMQ自动发现节点并加入集群，不需要手动在加入集群（新版本官方镜像包里面自动加入了集群工具Rabbitmq-peer-discovery-k8s，3.7之前是rabbitmq-autocluste工具已被淘汰)
Github地址 https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s

# 进入rabbitmq-0节点查看集群状态
[root@master rabbitmq]# kubectl exec -it rabbitmq-0 -n rabbit-mq /bin/bash
root@rabbitmq-0:/# rabbitmqctl cluster_status
RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Cluster status of node rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local ...
Basics

Cluster name: rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local

Disk Nodes

rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-headless.rabbit-mq.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-headless.rabbit-mq.svc.cluster.local

Running Nodes

rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-headless.rabbit-mq.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-headless.rabbit-mq.svc.cluster.local

Versions

rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local: RabbitMQ 3.9.5 on Erlang 24.0.5
rabbit@rabbitmq-1.rabbitmq-headless.rabbit-mq.svc.cluster.local: RabbitMQ 3.9.5 on Erlang 24.0.5
rabbit@rabbitmq-2.rabbitmq-headless.rabbit-mq.svc.cluster.local: RabbitMQ 3.9.5 on Erlang 24.0.5

Maintenance status

Node: rabbit@rabbitmq-0.rabbitmq-headless.rabbit-mq.svc.cluster.local, status: not under maintenance
Node: rabbit@rabbitmq-1.rabbitmq-headless.rabbit-mq.svc.cluster.local, status: not under maintenance
Node: rabbit@rabbitmq-2.rabbitmq-headless.rabbit-mq.svc.cluster.local, status: not under maintenance

Alarms

(none)

三.RabbitMQ Management 管理界面查看集群状态。

http://nodeip:35672

部署完成。

参考文档:

Github地址 https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s

Kubernetes 中完美部署RabbitMQ Cluster_coco3848的博客-CSDN博客

kubernetes使用rabbitmq-peer-discovery-k8s部署rabbitmq集群 - 灰信网（软件开发博客聚合）

k8s集群部署rabbitmq集群 - 大碗油泼 - 博客园

原文链接：https://blog.csdn.net/m0_57480266/article/details/120313269