Kubernetes部署Ceph StorageClass动态存储

官网文档:

Ceph Docs

Ceph Docs

 

先决条件

Rook 可以安装在任何现有的 Kubernetes 集群上,只要它满足最低版本并且 Rook 被授予所需的权限(有关更多信息,请参见下文)。

最低版本

Ceph 算子支持Kubernetes v1.16或更高版本。

Ceph 先决条件

为了配置 Ceph 存储集群,至少需要以下本地存储选项之一:

  • 原始设备(无分区或格式化文件系统)
  • 原始分区(无格式化文件系统)
  • block模式下存储类中可用的 PV

您可以使用以下命令确认您的分区或设备是否使用文件系统格式化。

lsblk -f
NAME                  FSTYPE      LABEL UUID                                   MOUNTPOINT
vda
└─vda1                LVM2_member       >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB
 ├─ubuntu--vg-root   ext4              c2366f76-6e21-4f10-a8f3-6776212e2fe4   /
 └─ubuntu--vg-swap_1 swap              9492a3dc-ad75-47cd-9596-678e8cf17ff9   [SWAP]
vdb

如果该FSTYPE字段不为空,则在相应设备的顶部有一个文件系统。在这个例子中,你可以使用vdbCeph 而不能使用vda它的分区

LVM 包

Ceph OSD 在以下场景中依赖于 LVM:

  • OSD 在原始设备或分区上创建
  • 如果启用了加密(encryptedDevice: true在集群 CR 中)
  • 指定了一个metadata设备

在这些情况下,OSD 不需要 LVM:

  • 使用 storageClassDeviceSets

如果您的方案需要 LVM,则 LVM 需要在将运行 OSD 的主机上可用。某些 Linux 发行版不附带该lvm2软件包。在你的 k8s 集群中的所有存储节点上都需要这个包来运行 Ceph OSD。如果没有这个包,即使 Rook 能够成功创建 Ceph OSD,当一个节点重新启动时,在重新启动的节点上运行的 OSD pod 将无法启动。请使用您的 Linux 发行版的包管理器安装 LVM。例如:

CentOS:

sudo yum install -y lvm2

Ubuntu:

sudo apt-get install -y lvm2

查看内核是否支持

# modprobe rbd

# lsmod | grep rbd

rbd                    94208  0

libceph               360448  1 rbd

 

克隆分支

$ git clone --single-branch --branch v1.8.2 https://github.com/rook/rook.git

部署 Rook Operator

cd rook-master/deploy/examples/
kubectl apply -f crds.yaml -f common.yaml -f operator.yaml


kubectl -n rook-ceph get pod

等待pod转为Running;确认方式如下;

[root@master01 ~]# kubectl get pods -n rook-ceph -l app=rook-ceph-operator
NAME                                  READY   STATUS    RESTARTS      AGE
rook-ceph-operator-757546f8c7-x2ft5   1/1     Running   1 (23h ago)   24h

官方提供了三个yaml文件:

  • cluster.yaml:在裸机上运行的生产集群的集群设置。至少需要三个工作节点。
  • cluster-on-pvc.yaml:在动态云环境中运行的生产集群的集群设置。
  • cluster-test.yaml:用于测试环境(例如 minikube)的集群设置。

创建 Ceph 集群

[root@master01 examples]# kubectl apply -f cluster.yaml 

等待pod转为Running;确认方式如下:

[root@master01 ~]# kubectl get pods -n rook-ceph
NAME                                               READY   STATUS      RESTARTS         AGE
csi-cephfsplugin-6pc7x                             3/3     Running     3 (23h ago)      24h
csi-cephfsplugin-nzt76                             3/3     Running     3 (23h ago)      24h
csi-cephfsplugin-provisioner-689686b44-pr89g       6/6     Running     10 (8m15s ago)   24h
csi-cephfsplugin-provisioner-689686b44-wwhnn       6/6     Running     6 (23h ago)      24h
csi-cephfsplugin-sk6t7                             3/3     Running     3 (23h ago)      24h
csi-rbdplugin-c9mx9                                3/3     Running     3 (23h ago)      24h
csi-rbdplugin-dkspl                                3/3     Running     3 (23h ago)      24h
csi-rbdplugin-f4ckl                                3/3     Running     3 (23h ago)      24h
csi-rbdplugin-provisioner-5775fb866b-6k287         6/6     Running     6 (23h ago)      24h
csi-rbdplugin-provisioner-5775fb866b-mqkzz         6/6     Running     10 (8m7s ago)    24h
rook-ceph-crashcollector-node01-689c89b6b5-fvk5s   1/1     Running     1 (23h ago)      24h
rook-ceph-crashcollector-node02-7b5c5ff89d-qf9wv   1/1     Running     0                23h
rook-ceph-crashcollector-node03-79b668ff7f-4r8qk   1/1     Running     1 (23h ago)      23h
rook-ceph-mgr-a-588db7c758-wwvqd                   1/1     Running     1 (23h ago)      24h
rook-ceph-mon-c-9b5c4d667-hn65f                    1/1     Running     2 (23h ago)      24h
rook-ceph-mon-p-5c44dcc75c-jx7dv                   1/1     Running     0                23h
rook-ceph-mon-s-945b47db4-kr8mr                    1/1     Running     0                23h
rook-ceph-operator-757546f8c7-x2ft5                1/1     Running     1 (23h ago)      25h
rook-ceph-osd-0-5dc59d59c9-d2qc9                   1/1     Running     0                23h
rook-ceph-osd-1-58bcb4d94-x6zkj                    1/1     Running     0                23h
rook-ceph-osd-2-86bd47f655-pcqp4                   1/1     Running     0                23h
rook-ceph-osd-prepare-node01--1-x2w6h              0/1     Completed   0                23h
rook-ceph-osd-prepare-node02--1-72d59              0/1     Completed   0                23h
rook-ceph-osd-prepare-node03--1-xshd4              0/1     Completed   0                23h
rook-ceph-tools-555c879675-ksbkm                   1/1     Running     0                23h

有些镜像源是谷歌的,拉取不下来,可以将镜像源改为阿里云的;

将镜像中的 k8s.gcr.io 替换为 registry.aliyuncs.com/google_containers 

例如:
docker image pull k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0   改为docker image pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0

镜像拉取下来后,重新打个标签;

docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0

要验证集群是否处于健康状态,请连接到Rook 工具箱并运行 ceph status命令。

交互式工具箱

运行 rook-ceph-tools pod:

kubectl create -f deploy/examples/toolbox.yaml

等待pod转为Running;

[root@master01 ~]# kubectl -n rook-ceph rollout status deploy/rook-ceph-tools
deployment "rook-ceph-tools" successfully rolled out

pod运行起来后,可以使用如下命令进入容器:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

工具箱中的所有可用工具都可以满足您的故障排除需求。

示例

  • ceph status
  • ceph osd status
  • ceph df
  • rados df

比如现在我们要查看集群的状态,需要满足下面的条件才认为是健康的:

  • 所有 mons 应该达到法定数量
  • mgr 应该是激活状态
  • 至少有一个 OSD 处于激活状态
  • 如果不是 HEALTH_OK 状态,则应该查看告警或者错误信息

实操部分:

[root@master01 ~]# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph status
  cluster:
    id:     4ad33c5c-8d3e-406d-b0fe-3af02cce3718
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum c,p,s (age 45s)
    mgr: a(active, since 33s)
    osd: 3 osds: 3 up (since 37s), 3 in (since 23h)
 
  data:
    pools:   2 pools, 33 pgs
    objects: 134 objects, 389 MiB
    usage:   1.1 GiB used, 599 GiB / 600 GiB avail
    pgs:     33 active+clean
 
  io:
    client:   616 KiB/s wr, 0 op/s rd, 0 op/s wr
 
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph osd status
ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  node01   396M   199G      0        0       0        0   exists,up  
 1  node02   394M   199G      0        0       0        0   exists,up  
 2  node03   396M   199G      0        0       0        0   exists,up  
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    600 GiB  599 GiB  1.2 GiB   1.2 GiB       0.19
TOTAL  600 GiB  599 GiB  1.2 GiB   1.2 GiB       0.19
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
replicapool             1   32  385 MiB      136  1.1 GiB   0.20    190 GiB
device_health_metrics   2    1      0 B        0      0 B      0    190 GiB
[rook@rook-ceph-tools-555c879675-cckc4 /]$ rados df
POOL_NAME                 USED  OBJECTS  CLONES  COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS       RD  WR_OPS       WR  USED COMPR  UNDER COMPR
device_health_metrics      0 B        0       0       0                   0        0         0       0      0 B       0      0 B         0 B          0 B
replicapool            1.1 GiB      136       0     408                   0        0         0     846  6.8 MiB    1054  396 MiB         0 B          0 B

total_objects    136
total_used       1.2 GiB
total_avail      599 GiB
total_space      600 GiB

接下来我们该如何使用ceph存储呢?

配置动态持久化存储

在 Rook 可以配置存储之前,需要创建一个StorageClass和。CephBlockPool这将允许 Kubernetes 在配置持久卷时与 Rook 进行互操作。

注意:此示例要求每个节点至少有 1 个 OSD,每个 OSD 位于3 个不同的节点上。

每个 OSD 必须位于不同的节点上,因为failureDomain设置为hostreplicated.size设置为3

因为我们已经把仓库克隆至本地了,所以我们直接进入相应目录,执行yaml文件即可,您也可以根据自己的需求来对yaml文件进行修改;

[root@master01 ~]# kubectl apply -f rook-master/deploy/examples/csi/rbd/storageclass.yaml 

yaml文件如下:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    # clusterID is the namespace where the rook cluster is running
    clusterID: rook-ceph
    # Ceph pool into which the RBD image shall be created
    pool: replicapool

    # (optional) mapOptions is a comma-separated list of map options.
    # For krbd options refer
    # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
    # For nbd options refer
    # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
    # mapOptions: lock_on_read,queue_depth=1024

    # (optional) unmapOptions is a comma-separated list of unmap options.
    # For krbd options refer
    # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
    # For nbd options refer
    # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
    # unmapOptions: force

    # RBD image format. Defaults to "2".
    imageFormat: "2"

    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
    imageFeatures: layering

    # The secrets contain Ceph admin credentials.
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

    # Specify the filesystem type of the volume. If not specified, csi-provisioner
    # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
    # in hyperconverged settings where the volume is mounted on the same node as the osds.
    csi.storage.k8s.io/fstype: ext4

# Delete the rbd volume when a PVC is deleted
reclaimPolicy: Delete

# Optional, if you want to add dynamic resize for PVC. Works for Kubernetes 1.14+
# For now only ext3, ext4, xfs resize support provided, like in Kubernetes itself.
allowVolumeExpansion: true

如果您在“rook-ceph”以外的命名空间中部署了 Rook 运算符,请更改配置器中的前缀以匹配您使用的命名空间。例如,如果 Rook 操作符在命名空间“my-namespace”中运行,则配置器值应该是“my-namespace.rbd.csi.ceph.com”。

 使用存储:Wordpress 示例(官方示例)

[root@master01 ~]# cd rook-master/deploy/examples/
[root@master01 examples]# kubectl apply mysql.yaml 
[root@master01 examples]# kubectl apply wordpress.yaml

查看pvc;

[root@master01 examples]# kubectl get pvc 
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-1ca21260-380b-4a4d-90e8-e2908391826b   1Gi        RWO            rook-ceph-block   24h
wp-pv-claim      Bound    pvc-5d756ef2-5909-4b38-b2f7-3c4b8ae23e8d   1Gi        RWO            rook-ceph-block   2s

查看pv;(pv是动态自己创建的)

[root@master01 examples]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS      REASON   AGE
pvc-1ca21260-380b-4a4d-90e8-e2908391826b   1Gi        RWO            Delete           Bound    default/mysql-pv-claim    rook-ceph-block            24h
pvc-28c4cac5-6dc6-4e6d-aa14-8ced1ccd988e   10Gi       RWO            Delete           Bound    jenkins/jenkins-pvc       rook-ceph-block            23h
pvc-5d756ef2-5909-4b38-b2f7-3c4b8ae23e8d   1Gi        RWO            Delete           Bound    default/wp-pv-claim       rook-ceph-block            34s
pvc-704293c0-6489-4512-870e-d6b2ef4ed86a   10Gi       RWO            Delete           Bound    jenkins/pvc-maven-cache   rook-ceph-block            22h


版权声明:本文为weixin_45588777原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接和本声明。