Ceph

介绍

一个 Ceph 存储集群至少需要一个 Ceph Monitor、Ceph Manager 和 Ceph OSD（对象存储守护进程）。运行 Ceph 文件系统客户端时也需要 Ceph 元数据服务器。

Monitors：Ceph Monitor ( ceph-mon) 维护集群状态的映射，包括监视器映射、管理器映射、OSD 映射、MDS 映射和 CRUSH 映射。这些映射是 Ceph 守护进程相互协调所需的关键集群状态。监视器还负责管理守护进程和客户端之间的身份验证。冗余和高可用性通常需要至少三个监视器。

Managers：Ceph Manager守护进程 ( ceph-mgr) 负责跟踪运行时指标和 Ceph 集群的当前状态，包括存储利用率、当前性能指标和系统负载。Ceph Manager 守护进程还托管基于 python 的模块来管理和公开 Ceph 集群信息，包括基于 Web 的Ceph Dashboard和 REST API。高可用性通常需要至少两个管理器。

Ceph OSD：一个Ceph OSD（对象存储守护进程 ceph-osd）存储数据、处理数据复制、恢复、重新平衡，并通过检查其他 Ceph OSD 守护进程的心跳来向 Ceph 监视器和管理器提供一些监视信息。冗余和高可用性通常需要至少 3 个 Ceph OSD。

MDS：Ceph 元数据服务器(MDS ceph-mds) 代表Ceph 文件系统存储元数据（即 Ceph 块设备和 Ceph 对象存储不使用 MDS）。Ceph 元数据服务器允许 POSIX 文件系统用户执行基本命令（如 ls、find等），而不会给 Ceph 存储集群带来巨大负担。

CEPH 存储集群

Ceph 存储集群是所有 Ceph 部署的基础。基于RADOS，Ceph 存储集群由几种类型的守护进程组成：

1、Ceph OSD 守护进程(OSD) 将数据作为对象存储在存储节点上

2、Ceph Monitor (MON) 维护集群映射的主副本。

3、Ceph Manager 管理器 守护进程

一个 Ceph 存储集群可能包含数千个存储节点。一个最小的系统至少有一个 Ceph Monitor 和两个 Ceph OSD Daemons 用于数据复制。

Ceph 文件系统、Ceph 对象存储和 Ceph 块设备从 Ceph 存储集群读取数据并将数据写入到 Ceph 存储集群。

CEPH 文件系统

Ceph 文件系统或CephFS是一个符合 POSIX 的文件系统，构建在 Ceph 的分布式对象存储RADOS之上。CephFS 致力于为各种应用程序（包括共享主目录、HPC 暂存空间和分布式工作流共享存储等传统用例）提供最先进的、多用途、高可用性和高性能的文件存储。

CephFS 通过使用一些新颖的架构选择来实现这些目标。值得注意的是，文件元数据与文件数据存储在单独的 RADOS 池中，并通过可调整大小的元数据服务器集群或MDS提供服务，该集群可以扩展以支持更高吞吐量的元数据工作负载。文件系统的客户端可以直接访问 RADOS 以读取和写入文件数据块。出于这个原因，工作负载可能会随着底层 RADOS 对象存储的大小而线性扩展；也就是说，没有网关或代理为客户端调解数据 I/O。

对数据的访问是通过 MDS 集群来协调的，MDS 集群作为由客户端和 MDS 共同维护的分布式元数据缓存状态的权限。元数据的突变由每个 MDS 聚合成一系列有效的写入 RADOS 上的日志；MDS 没有在本地存储元数据状态。该模型允许在 POSIX 文件系统的上下文中客户端之间进行一致和快速的协作。

集群安装

参考视频安装

添加硬盘及内外网

修改virbr0

vim /etc/libvirt/qemu/networks/default.xml

vim /etc/hosts

其余主机配置跟随以上图片。。。。

在ceph-admin节点上安装ceph及epel的yum仓库配置

rpm -ivh https://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ceph-deploy-1.5.29-0.noarch.rpm
yum install epel-release

个节点创建用户、密码及用户权限

useradd cephadm && echo 123456 |passwd --stdin cephadm

vim /etc/sudoers.d/cephadm 添加以下内容
cephadm ALL=(root) NOPASSWD: ALL

密钥配置

cephadm 用户下：ssh-keygen -t rsa -P ‘’

复制到自己的用户下
复制到其余主机下

[cephadm@ceph-client ~]$ scp -rp .ssh/ cephadm@mon01:/home/cephadm
[cephadm@ceph-client ~]$ scp -rp .ssh/ cephadm@mon02:/home/cephadm
[cephadm@ceph-client ~]$ scp -rp .ssh/ cephadm@mon03:/home/cephadm
[cephadm@ceph-client ~]$ scp -rp .ssh/ cephadm@stor04:/home/cephadm

设置权限到各主机到cephadm用户下

[cephadm@ceph-client ~]$ sudo scp /etc/sudoers.d/cephadm cephadm@mon01/etc/sudoers.d/
[cephadm@ceph-client ~]$ sudo scp /etc/sudoers.d/cephadm cephadm@mon02/etc/sudoers.d/
[cephadm@ceph-client ~]$ sudo scp /etc/sudoers.d/cephadm cephadm@mon03/etc/sudoers.d/
[cephadm@ceph-client ~]$ sudo scp /etc/sudoers.d/cephadm cephadm@stor04/etc/sudoers.d/

测试

设置权限到各主机到root用户下

[root@ceph-client ~]# scp -rp /etc/sudoers.d/cephadm mon01:/etc/sudoers.d/
[root@ceph-client ~]# scp -rp /etc/sudoers.d/cephadm mon02:/etc/sudoers.d/
[root@ceph-client ~]# scp -rp /etc/sudoers.d/cephadm mon03:/etc/sudoers.d/
[root@ceph-client ~]# scp -rp /etc/sudoers.d/cephadm stor04:/etc/sudoers.d/

在管理节点安装ceph-deploy

[root@ceph-client ~]# yum install ceph-deploy python-setuptools python2-subprocess32

####5、部署RADOS存储集群
首先在管理节点上以cephadm用户创建集群相关的配置文件目录:

[cephadm@ceph-client ~]$ mkdir ceph-cluster
[cephadm@ceph-client ~]$ cd ceph-cluster

初始化第一个mon节点

[cephadm@ceph-client ceph-cluster]$ 
ceph-deploy new --cluster-network 192.168.122.0/16 --public_network 192.168.219.0/16 mon01

配置Ceph集群节点

[cephadm@ceph-client ceph-cluster]$ ceph-deploy install --no-adjust-repos mon01 monn02 mon03 stor04

配置初始MON节点，并收集所有密钥:

[cephadm@ceph-client ceph-cluster]$ ceph-deploy mon create-initial

mon节点下

ps -aux|grep ceph-mon

把配置文件和admin密钥拷贝Ceph集群各节点，以免得每次执行"ceph"命令行时不得不明确指定MON节点地址和ceph.client.admin.keyring:

[cephadm@ceph-client ceph-cluster]$ ceph-deploy admin ceph-admin mon01 mon02 mon03 stor04

配置Manager节点，启动ceph-mgr进程

[cephadm@ceph-client ceph-cluster]$ ceph-deploy mgr create stor04
[cephadm@ceph-client ceph-cluster]$ ceph-deploy mgr create mon03

在stor04和mon03 查询是否成功
ps -aux|grep ceph-mgr

在root用户下设置密钥权限

添加osd
清空磁盘

[cephadm@ceph-client ceph-cluster]$ ceph-deploy disk zap mon01 /dev/sdb
[cephadm@ceph-client ceph-cluster]$ ceph-deploy disk zap mon01 /dev/sdc

添加osd

[cephadm@ceph-client ceph-cluster]$  ceph-deploy osd create mon01 --data /dev/sdb
[cephadm@ceph-client ceph-cluster]$  ceph-deploy osd create mon01 --data /dev/sdc

根据以上添加6个osd

ceph的文件存储CephFs

参考博客

创建文件系统

一个 Ceph 文件系统需要至少两个 RADOS 存储池，一个用于数据、一个用于元数据。配置这些存储池时需考虑：

ceph osd pool create cephfs_data 128
ceph osd pool create cephfs_metadata 128

创建文件系统

ceph fs new cephfs cephfs_metadata cephfs_data

使用下面的命令可以查看创建的CephFS

ceph fs ls

挂载CephFs

根据monitor监视器，IP为192.168.219.106。使用mount命令挂载到mycephfs，如果启用了cephx认证的Ceph文件系统，挂载时我们必须指定用户名、密钥。secret在/etc/ceph/ceph.client.admin.keyring可以查到。

sudo mkdir /mycephfs
sudo mount -t ceph 192.168.219.106:6789:/  /mycephfs -o name=admin,secret=AQAXRzdiR03EKhAAiuVepymHpRLkEsOqtKlI2A==

cephfs微服务开发

参考博客

linux环境配置

在开发机安装libcephfs相关开发包

通过rpm -qa|grep ceph 查看

缺少libcephfs_jni1 yun install libcephfs_jni1

设置链接：

ln -s /usr/lib64/libcephfs_jni.so.1.0.0 /usr/lib/libcephfs_jni.so.1
ln -s /usr/lib64/libcephfs_jni.so.1.0.0 /usr/lib/libcephfs_jni.so

同时需要安装ceph-common组件：yum -y install ceph-common

目录结构

maven依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.5.6</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.zxh</groupId>
    <artifactId>ceph</artifactId>
    <version>1.0-SNAPSHOT</version>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>com.ceph</groupId>
        <artifactId>libcephfs</artifactId>
        <version>0.80.5</version>
    </dependency>
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger2</artifactId>
        <version>2.9.2</version>
    </dependency>
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger-ui</artifactId>
        <version>2.9.2</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.25</version>
    </dependency>
</dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

cephfs挂载与目录基本方法实现以及上传下载

Service代码：以CephMount类为主要操作类，实现cephfs的操作

package com.zxh;
 
import com.ceph.fs.CephMount;
import com.ceph.fs.CephStat;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.http.HttpServletResponse;
import java.io.*;

@Service
public class CephfsServiceImpl  {
    Logger logger= LogManager.getLogger(CephfsServiceImpl.class);
    private CephMount mount=null;


    public Boolean mountCephfsByRoot(){
        try {
            this.mount = new CephMount("admin");
            this.mount.conf_set("mon_host", "192.168.219.106");
            //System.out.println(mount.conf_get("mon_host"));
            this.mount.conf_set("key","AQAXRzdiR03EKhAAiuVepymHpRLkEsOqtKlI2A==");
            this.mount.mount("/");
            return true;
        }catch (Exception e){
            e.printStackTrace();
        }
        return false;
    }
 
 

    public String[] createDirByPath(String path){
        String[] dirList = null;
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.mkdirs(path, 0777);
            dirList = this.mount.listdir("/");
            return dirList;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 

    public String[] deleteDirByPath(String path){
        String[] dirList = null;
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.rmdir(path);
            dirList = this.mount.listdir("/");
            return dirList;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 

    public CephStat getFileStatusByPath(String path){
        CephStat stat = new CephStat();
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.lstat(path, stat);
            return stat;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 

    public String readFileByPath(String path){
        CephStat stat = new CephStat();
        String context=null;
        try {
            if (this.mount == null){
                return null;
            }
            int fd = this.mount.open(path, CephMount.O_RDONLY, 0);
            this.mount.fstat(fd, stat);
            byte[] buffer = new byte[(int)stat.size];
            this.mount.read(fd, buffer, stat.size, 0);
            context = new String(buffer);
            this.mount.close(fd);
            return context;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }

    public Boolean uploadFileByPath(String filePath, String fileName){

        // exit with null if not mount
        if (this.mount == null){
            logger.info("Ceph fs not mount!");
            return null;
        }

        // file definition
        char pathChar = File.separatorChar;
        String fileFullName = "";
        Long fileLength = 0l;
        Long uploadedLength = 0l;
        File file = null;

        // Io
        FileInputStream fis = null;

        // get local file info
        fileFullName = filePath + pathChar + fileName;
        logger.info("fileInfo:{}",fileFullName);
        file = new File(fileFullName);
        if (!file.exists()){
            logger.info("File not exist!");
            return false;
        }
        fileLength = file.length();

        // get io from local file
        try {
            fis = new FileInputStream(file);
        }catch (FileNotFoundException e){
            logger.info("Read local file failed!");
            e.printStackTrace();
        }

        // if file exists or not
        String[] dirList = null;
        Boolean fileExist = false;
        try {
            dirList = this.mount.listdir("/");
            for (String fileInfo : dirList){
                logger.info("fileInfo:{}",dirList);
                if (fileInfo.equals(fileName)){
                    fileExist = true;
                }
            }
        }catch (FileNotFoundException e){
            logger.info("File Path not exist!");
            e.printStackTrace();
        }

        // transfer file by diff pattern
        if (!fileExist){
            try {
                // create file and set mode WRITE
                this.mount.open(fileName, CephMount.O_CREAT, 0);
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);

                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                while ((length = fis.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);

                    // update length
                    uploadedLength += length;

                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);

                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("文件传输成功！");

                // chmod
                this.mount.fchmod(fd, 0666);

                // close
                this.mount.close(fd);
                if (fis != null){
                    fis.close();
                }
                return true;
            }catch (Exception e){
                logger.info("File transfer failed!");
                e.printStackTrace();
            }
        }else if (fileExist){
            try {
                // get file length
                CephStat stat = new CephStat();
                this.mount.stat(fileName, stat);
                uploadedLength = stat.size;
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);

                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                fis.skip(uploadedLength);
                while ((length = fis.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);

                    // update length
                    uploadedLength += length;

                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);

                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("断点文件传输成功！");

                // chmod
                this.mount.fchmod(fd, 0666);

                // close
                this.mount.close(fd);
                if (fis != null){
                    fis.close();
                }
                return true;
            }catch (Exception e){
                logger.info("BreakPoint transfer failed!");
                e.printStackTrace();
            }
        }else {
            try {
                if (fis != null){
                    fis.close();
                }
            }catch (Exception e){
                e.printStackTrace();
            }
            return false;
        }

        return false;
    }

    public Boolean downloadFileByPath(String filePath, String fileName) {

        // exit with null if not mount
        if (this.mount == null) {
            logger.info("Ceph fs not mount!");
            return null;
        }

        // file definition
        char pathChar = File.separatorChar;
        String fileFullName = "";
        Long fileLength = 0l;
        Long downloadedLength = 0l;
        File file = null;

        // IO
        FileOutputStream fos = null;
        RandomAccessFile raf = null;

        // new file object
        fileFullName = filePath + pathChar + fileName;
        file = new File(fileFullName);

        // get cephfs file size
        try {
            CephStat stat = new CephStat();
            this.mount.stat(fileName, stat);
            fileLength = stat.size;
        } catch (Exception e) {
            logger.info("Fail to get file size.");
            e.printStackTrace();
        }

        if (fileLength != 0) {
            if (!file.exists()) {
                // download file
                int length = 10240;
                byte[] bytes = new byte[length];
                try {
                    int fd = this.mount.open(fileName, CephMount.O_RDONLY, 0);
                    fos = new FileOutputStream(file);
                    float rate = 0;
                    String rateValue = "";
                    while ((fileLength - downloadedLength) >= length && (this.mount.read(fd, bytes, (long) length, downloadedLength)) != -1) {
                        fos.write(bytes, 0, length);
                        fos.flush();
                        downloadedLength += (long) length;

                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);

                        if (downloadedLength == fileLength) {
                            break;
                        }
                    }
                    if (downloadedLength != fileLength) {
                        this.mount.read(fd, bytes, fileLength - downloadedLength, downloadedLength);
                        fos.write(bytes, 0, (int) (fileLength - downloadedLength));
                        fos.flush();
                        downloadedLength = fileLength;

                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);
                    }

                    System.out.println("Download Success!");
                    fos.close();
                    this.mount.close(fd);
                    return true;
                } catch (Exception e) {
                    logger.info("First download fail!");
                    e.printStackTrace();
                }
            } else if (file.exists()) {
                // download file
                int length = 10240;
                byte[] bytes = new byte[length];
                Long filePoint = file.length();
                try {
                    int fd = this.mount.open(fileName, CephMount.O_RDONLY, 0);
                    raf = new RandomAccessFile(file, "rw");
                    raf.seek(filePoint);
                    downloadedLength = filePoint;
                    float rate = 0;
                    String rateValue = "";
                    while ((fileLength - downloadedLength) >= length && (this.mount.read(fd, bytes, (long) length, downloadedLength)) != -1) {
                        raf.write(bytes, 0, length);
                        downloadedLength += (long) length;

                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);

                        if (downloadedLength == fileLength) {
                            break;
                        }
                    }
                    if (downloadedLength != fileLength) {
                        this.mount.read(fd, bytes, fileLength - downloadedLength, downloadedLength);
                        raf.write(bytes, 0, (int) (fileLength - downloadedLength));
                        downloadedLength = fileLength;

                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);
                    }

                    System.out.println("Cut Point Download Success!");
                    raf.close();
                    this.mount.close(fd);
                    return true;
                } catch (Exception e) {
                    logger.info("Continue download fail!");
                    e.printStackTrace();
                }
            } else {
                logger.info("Unknown Error!");
                return false;
            }
        }
        return false;
    }

    public Boolean uploadFileByMultipart(MultipartFile uploadFile)  {
        String fileName=uploadFile.getOriginalFilename();
        InputStream inputStream = null;
        try {
            inputStream = uploadFile.getInputStream();
        } catch (IOException e) {
            e.printStackTrace();
        }
        // exit with null if not mount
        if (this.mount == null){
            logger.info("Ceph fs not mount!");
            return null;
        }

        // file definition
        char pathChar = File.separatorChar;
        String fileFullName = "";
        Long fileLength = 0l;
        fileLength=uploadFile.getSize();
        Long uploadedLength = 0l;

        // if file exists or not
        String[] dirList = null;
        Boolean fileExist = false;
        try {
            dirList = this.mount.listdir("/");
            for (String fileInfo : dirList){
                logger.info("fileInfo:{}",dirList);
                if (fileInfo.equals(fileName)){
                    fileExist = true;
                }
            }
        }catch (FileNotFoundException e){
            logger.info("File Path not exist!");
            e.printStackTrace();
        }

        // transfer file by diff pattern
        if (!fileExist){
            try {
                // create file and set mode WRITE
                this.mount.open(fileName, CephMount.O_CREAT, 0);
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);

                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                while ((length = inputStream.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);

                    // update length
                    uploadedLength += length;

                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);

                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("文件传输成功！");

                // chmod
                this.mount.fchmod(fd, 0666);

                // close
                this.mount.close(fd);
                if (inputStream != null){
                    inputStream.close();
                }
                return true;
            }catch (Exception e){
                logger.info("File transfer failed!");
                e.printStackTrace();
            }
        }else if (fileExist){
            try {
                // get file length
                CephStat stat = new CephStat();
                this.mount.stat(fileName, stat);
                uploadedLength = stat.size;
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);

                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                inputStream.skip(uploadedLength);
                while ((length = inputStream.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);

                    // update length
                    uploadedLength += length;

                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);

                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("断点文件传输成功！");

                // chmod
                this.mount.fchmod(fd, 0666);

                // close
                this.mount.close(fd);
                if (inputStream != null){
                    inputStream.close();
                }
                return true;
            }catch (Exception e){
                logger.info("BreakPoint transfer failed!");
                e.printStackTrace();
            }
        }else {
            try {
                if (inputStream != null){
                    inputStream.close();
                }
            }catch (Exception e){
                e.printStackTrace();
            }
            return false;
        }
        return false;
    }


    public void downloadFile(String fileName, HttpServletResponse resp) {
        resp.setHeader("content-type", "application/octet-stream");
        resp.setContentType("application/octet-stream");
        resp.setHeader("Content-Disposition", "attachment;filename=" + fileName);
        // exit with null if not mount
        if (this.mount == null) {
            logger.info("Ceph fs not mount!");
            return ;
        }

        // file definition
        Long fileLength = 0l;
        Long downloadedLength = 0l;


        // IO
        OutputStream os = null;
        RandomAccessFile raf = null;
        try {
            os=resp.getOutputStream();
        } catch (IOException e) {
            e.printStackTrace();
        }

        // get cephfs file size
        try {
            CephStat stat = new CephStat();
            this.mount.stat(fileName, stat);
            fileLength = stat.size;
        } catch (Exception e) {
            logger.info("Fail to get file size.");
            e.printStackTrace();
        }

        if (fileLength != 0) {

                // download file
                int length = 10240;
                byte[] bytes = new byte[length];
                try {
                    int fd = this.mount.open(fileName, CephMount.O_RDONLY, 0);
                    float rate = 0;
                    String rateValue = "";
                    while ((fileLength - downloadedLength) >= length && (this.mount.read(fd, bytes, (long) length, downloadedLength)) != -1) {
                        os.write(bytes, 0, length);
                        os.flush();
                        downloadedLength += (long) length;

                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);

                        if (downloadedLength == fileLength) {
                            break;
                        }
                    }
                    if (downloadedLength != fileLength) {
                        this.mount.read(fd, bytes, fileLength - downloadedLength, downloadedLength);
                        os.write(bytes, 0, (int) (fileLength - downloadedLength));
                        os.flush();
                        downloadedLength = fileLength;
                        // output transfer rate
                        rate = (float) downloadedLength * 100 / (float) fileLength;
                        rateValue = (int) rate + "%";
                        System.out.println(rateValue);
                    }

                    System.out.println("Download Success!");
                    os.close();
                    this.mount.close(fd);
                    return ;
                } catch (Exception e) {
                    logger.info("First download fail!");
                    e.printStackTrace();
                }
        }
    }
}

控制类+swagger实现api调用接口：

package com.zxh;


import com.ceph.fs.CephStat;
import io.swagger.annotations.ApiOperation;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.http.HttpServletResponse;

@RestController
@RequestMapping(value = "/cephfs/api")
public class CephfsController {
 
    private CephfsServiceImpl cephfsService = new CephfsServiceImpl();
 
    @ApiOperation(value = "Mount", notes = "Mount")
    @RequestMapping(value = "/mount", method = RequestMethod.GET)
    public Boolean mountCephFsByRoot(){
        return cephfsService.mountCephfsByRoot();
    }
 
    @ApiOperation(value = "CreateDir", notes = "CreateDir")
    @RequestMapping(value = "/createdir", method = RequestMethod.POST)
    public String[] createDirByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.createDirByPath(path);
    }
 
    @ApiOperation(value = "DeleteDir", notes = "DeleteDir")
    @RequestMapping(value = "/deletedir", method = RequestMethod.DELETE)
    public String[] deleteDirByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.deleteDirByPath(path);
    }
 
    @ApiOperation(value = "FileStatus", notes = "FileStatus")
    @RequestMapping(value = "/getfilestatus", method = RequestMethod.GET)
    public CephStat getFileStatusByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.getFileStatusByPath(path);
    }

    @ApiOperation(value = "FileContext", notes = "FileContext")
    @RequestMapping(value = "/getfilecontext", method = RequestMethod.GET)
    public String readFileByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.readFileByPath(path);
    }

    @ApiOperation(value = "UploadFile",notes = "UploadFile")
    @RequestMapping(value = "/uploadfile", method = RequestMethod.GET)
    public boolean uploadFileByPath (String filePath, String fileName){
        return cephfsService.uploadFileByPath(filePath,fileName);
    }
    @ApiOperation(value = "DownloadFile",notes = "DownloadFile")
    @RequestMapping(value = "/downloadfile", method = RequestMethod.GET)
    public Boolean downloadFileByPath(String filePath, String fileName){
        return cephfsService.downloadFileByPath(filePath,fileName);
    }
    @ApiOperation(value = "UploadFile2",notes = "UploadFile2")
    @RequestMapping(value = "/uploadfile2", method = RequestMethod.POST)
    public Boolean uploadFileByMultipart(MultipartFile uploadFile){
        return cephfsService.uploadFileByMultipart(uploadFile);
    }
    @ApiOperation(value = "DownloadFile",notes = "DownloadFile")
    @RequestMapping(value = "/downloadfile2", method = RequestMethod.GET)
    public void downloadFile(String fileName, HttpServletResponse res) {
        cephfsService.downloadFile(fileName,res);
    }
}

Swagger配置

package com.zxh;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import springfox.documentation.service.ApiInfo;
import springfox.documentation.service.Contact;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;

import java.util.ArrayList;

@Configuration
@EnableSwagger2
public class SwaggerConfig {
    @Bean
    public Docket decket(){
        return new Docket(DocumentationType.SWAGGER_2)
                .apiInfo(new ApiInfo("Api Documentation", "Api Documentation", "1.0", "urn:tos", new Contact("","",""), "Apache 2.0", "http://www.apache.org/licenses/LICENSE-2.0", new ArrayList()));
    }
}

application.yml

spring:
  servlet:
    multipart:
      max-file-size: 500MB
      max-request-size: 500MB
server:
  port: 8080

测试：

linux环境运行jar包

进入mycephfs

 cd /mycephfs/
 [root@ceph-client ~]# ll /mycephfs/
总用量 0

打开http://192.168.219.105:8080/swagger-ui.html进行测试


执行mount接口
上传


下载