官网文档:
先决条件
Rook 可以安装在任何现有的 Kubernetes 集群上,只要它满足最低版本并且 Rook 被授予所需的权限(有关更多信息,请参见下文)。
最低版本
Ceph 算子支持Kubernetes v1.16或更高版本。
Ceph 先决条件
为了配置 Ceph 存储集群,至少需要以下本地存储选项之一:
- 原始设备(无分区或格式化文件系统)
- 原始分区(无格式化文件系统)
block
模式下存储类中可用的 PV
您可以使用以下命令确认您的分区或设备是否使用文件系统格式化。
lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT vda └─vda1 LVM2_member >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB ├─ubuntu--vg-root ext4 c2366f76-6e21-4f10-a8f3-6776212e2fe4 / └─ubuntu--vg-swap_1 swap 9492a3dc-ad75-47cd-9596-678e8cf17ff9 [SWAP] vdb
如果该FSTYPE
字段不为空,则在相应设备的顶部有一个文件系统。在这个例子中,你可以使用vdb
Ceph 而不能使用vda
它的分区
LVM 包
Ceph OSD 在以下场景中依赖于 LVM:
- OSD 在原始设备或分区上创建
- 如果启用了加密(
encryptedDevice: true
在集群 CR 中) - 指定了一个
metadata
设备
在这些情况下,OSD 不需要 LVM:
- 使用
storageClassDeviceSets
如果您的方案需要 LVM,则 LVM 需要在将运行 OSD 的主机上可用。某些 Linux 发行版不附带该lvm2
软件包。在你的 k8s 集群中的所有存储节点上都需要这个包来运行 Ceph OSD。如果没有这个包,即使 Rook 能够成功创建 Ceph OSD,当一个节点重新启动时,在重新启动的节点上运行的 OSD pod 将无法启动。请使用您的 Linux 发行版的包管理器安装 LVM。例如:
CentOS:
sudo yum install -y lvm2
Ubuntu:
sudo apt-get install -y lvm2
查看内核是否支持
# modprobe rbd
# lsmod | grep rbd
rbd 94208 0
libceph 360448 1 rbd
克隆分支
$ git clone --single-branch --branch v1.8.2 https://github.com/rook/rook.git
部署 Rook Operator
cd rook-master/deploy/examples/
kubectl apply -f crds.yaml -f common.yaml -f operator.yaml
kubectl -n rook-ceph get pod
等待pod转为Running;确认方式如下;
[root@master01 ~]# kubectl get pods -n rook-ceph -l app=rook-ceph-operator
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-757546f8c7-x2ft5 1/1 Running 1 (23h ago) 24h
官方提供了三个yaml文件:
- cluster.yaml:在裸机上运行的生产集群的集群设置。至少需要三个工作节点。
- cluster-on-pvc.yaml:在动态云环境中运行的生产集群的集群设置。
- cluster-test.yaml:用于测试环境(例如 minikube)的集群设置。
创建 Ceph 集群
[root@master01 examples]# kubectl apply -f cluster.yaml
等待pod转为Running;确认方式如下:
[root@master01 ~]# kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-6pc7x 3/3 Running 3 (23h ago) 24h
csi-cephfsplugin-nzt76 3/3 Running 3 (23h ago) 24h
csi-cephfsplugin-provisioner-689686b44-pr89g 6/6 Running 10 (8m15s ago) 24h
csi-cephfsplugin-provisioner-689686b44-wwhnn 6/6 Running 6 (23h ago) 24h
csi-cephfsplugin-sk6t7 3/3 Running 3 (23h ago) 24h
csi-rbdplugin-c9mx9 3/3 Running 3 (23h ago) 24h
csi-rbdplugin-dkspl 3/3 Running 3 (23h ago) 24h
csi-rbdplugin-f4ckl 3/3 Running 3 (23h ago) 24h
csi-rbdplugin-provisioner-5775fb866b-6k287 6/6 Running 6 (23h ago) 24h
csi-rbdplugin-provisioner-5775fb866b-mqkzz 6/6 Running 10 (8m7s ago) 24h
rook-ceph-crashcollector-node01-689c89b6b5-fvk5s 1/1 Running 1 (23h ago) 24h
rook-ceph-crashcollector-node02-7b5c5ff89d-qf9wv 1/1 Running 0 23h
rook-ceph-crashcollector-node03-79b668ff7f-4r8qk 1/1 Running 1 (23h ago) 23h
rook-ceph-mgr-a-588db7c758-wwvqd 1/1 Running 1 (23h ago) 24h
rook-ceph-mon-c-9b5c4d667-hn65f 1/1 Running 2 (23h ago) 24h
rook-ceph-mon-p-5c44dcc75c-jx7dv 1/1 Running 0 23h
rook-ceph-mon-s-945b47db4-kr8mr 1/1 Running 0 23h
rook-ceph-operator-757546f8c7-x2ft5 1/1 Running 1 (23h ago) 25h
rook-ceph-osd-0-5dc59d59c9-d2qc9 1/1 Running 0 23h
rook-ceph-osd-1-58bcb4d94-x6zkj 1/1 Running 0 23h
rook-ceph-osd-2-86bd47f655-pcqp4 1/1 Running 0 23h
rook-ceph-osd-prepare-node01--1-x2w6h 0/1 Completed 0 23h
rook-ceph-osd-prepare-node02--1-72d59 0/1 Completed 0 23h
rook-ceph-osd-prepare-node03--1-xshd4 0/1 Completed 0 23h
rook-ceph-tools-555c879675-ksbkm 1/1 Running 0 23h
有些镜像源是谷歌的,拉取不下来,可以将镜像源改为阿里云的;
将镜像中的 k8s.gcr.io
替换为 registry.aliyuncs.com/google_containers
例如:
docker image pull k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0 改为docker image pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0
镜像拉取下来后,重新打个标签;
docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
要验证集群是否处于健康状态,请连接到Rook 工具箱并运行 ceph status
命令。
交互式工具箱
运行 rook-ceph-tools pod:
kubectl create -f deploy/examples/toolbox.yaml
等待pod转为Running;
[root@master01 ~]# kubectl -n rook-ceph rollout status deploy/rook-ceph-tools
deployment "rook-ceph-tools" successfully rolled out
pod运行起来后,可以使用如下命令进入容器:
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
工具箱中的所有可用工具都可以满足您的故障排除需求。
示例:
ceph status
ceph osd status
ceph df
rados df
比如现在我们要查看集群的状态,需要满足下面的条件才认为是健康的:
- 所有 mons 应该达到法定数量
- mgr 应该是激活状态
- 至少有一个 OSD 处于激活状态
- 如果不是 HEALTH_OK 状态,则应该查看告警或者错误信息
实操部分:
[root@master01 ~]# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph status
cluster:
id: 4ad33c5c-8d3e-406d-b0fe-3af02cce3718
health: HEALTH_OK
services:
mon: 3 daemons, quorum c,p,s (age 45s)
mgr: a(active, since 33s)
osd: 3 osds: 3 up (since 37s), 3 in (since 23h)
data:
pools: 2 pools, 33 pgs
objects: 134 objects, 389 MiB
usage: 1.1 GiB used, 599 GiB / 600 GiB avail
pgs: 33 active+clean
io:
client: 616 KiB/s wr, 0 op/s rd, 0 op/s wr
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node01 396M 199G 0 0 0 0 exists,up
1 node02 394M 199G 0 0 0 0 exists,up
2 node03 396M 199G 0 0 0 0 exists,up
[rook@rook-ceph-tools-555c879675-cckc4 /]$ ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 600 GiB 599 GiB 1.2 GiB 1.2 GiB 0.19
TOTAL 600 GiB 599 GiB 1.2 GiB 1.2 GiB 0.19
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
replicapool 1 32 385 MiB 136 1.1 GiB 0.20 190 GiB
device_health_metrics 2 1 0 B 0 0 B 0 190 GiB
[rook@rook-ceph-tools-555c879675-cckc4 /]$ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
device_health_metrics 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
replicapool 1.1 GiB 136 0 408 0 0 0 846 6.8 MiB 1054 396 MiB 0 B 0 B
total_objects 136
total_used 1.2 GiB
total_avail 599 GiB
total_space 600 GiB
接下来我们该如何使用ceph存储呢?
配置动态持久化存储
在 Rook 可以配置存储之前,需要创建一个StorageClass和。CephBlockPool这将允许 Kubernetes 在配置持久卷时与 Rook 进行互操作。
注意:此示例要求每个节点至少有 1 个 OSD,每个 OSD 位于3 个不同的节点上。
每个 OSD 必须位于不同的节点上,因为failureDomain设置为host
且replicated.size
设置为3
。
因为我们已经把仓库克隆至本地了,所以我们直接进入相应目录,执行yaml文件即可,您也可以根据自己的需求来对yaml文件进行修改;
[root@master01 ~]# kubectl apply -f rook-master/deploy/examples/csi/rbd/storageclass.yaml
yaml文件如下:
apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-block # Change "rook-ceph" provisioner prefix to match the operator namespace if needed provisioner: rook-ceph.rbd.csi.ceph.com parameters: # clusterID is the namespace where the rook cluster is running clusterID: rook-ceph # Ceph pool into which the RBD image shall be created pool: replicapool # (optional) mapOptions is a comma-separated list of map options. # For krbd options refer # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options # For nbd options refer # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options # mapOptions: lock_on_read,queue_depth=1024 # (optional) unmapOptions is a comma-separated list of unmap options. # For krbd options refer # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options # For nbd options refer # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options # unmapOptions: force # RBD image format. Defaults to "2". imageFormat: "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. imageFeatures: layering # The secrets contain Ceph admin credentials. csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock # in hyperconverged settings where the volume is mounted on the same node as the osds. csi.storage.k8s.io/fstype: ext4 # Delete the rbd volume when a PVC is deleted reclaimPolicy: Delete # Optional, if you want to add dynamic resize for PVC. Works for Kubernetes 1.14+ # For now only ext3, ext4, xfs resize support provided, like in Kubernetes itself. allowVolumeExpansion: true
如果您在“rook-ceph”以外的命名空间中部署了 Rook 运算符,请更改配置器中的前缀以匹配您使用的命名空间。例如,如果 Rook 操作符在命名空间“my-namespace”中运行,则配置器值应该是“my-namespace.rbd.csi.ceph.com”。
使用存储:Wordpress 示例(官方示例)
[root@master01 ~]# cd rook-master/deploy/examples/
[root@master01 examples]# kubectl apply mysql.yaml
[root@master01 examples]# kubectl apply wordpress.yaml
查看pvc;
[root@master01 examples]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-1ca21260-380b-4a4d-90e8-e2908391826b 1Gi RWO rook-ceph-block 24h
wp-pv-claim Bound pvc-5d756ef2-5909-4b38-b2f7-3c4b8ae23e8d 1Gi RWO rook-ceph-block 2s
查看pv;(pv是动态自己创建的)
[root@master01 examples]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-1ca21260-380b-4a4d-90e8-e2908391826b 1Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 24h
pvc-28c4cac5-6dc6-4e6d-aa14-8ced1ccd988e 10Gi RWO Delete Bound jenkins/jenkins-pvc rook-ceph-block 23h
pvc-5d756ef2-5909-4b38-b2f7-3c4b8ae23e8d 1Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 34s
pvc-704293c0-6489-4512-870e-d6b2ef4ed86a 10Gi RWO Delete Bound jenkins/pvc-maven-cache rook-ceph-block 22h