💻 工作存储CEPH2-部署
CEPH, CentOS7, ceph-deploy, 环境搭建
Source
Edit
History

如何使用 ceph-deploy 部署一个指定版本号的 ceph 集群

前言

在 上文 中我们使用 ceph-deploy 搭建 ceph 集群。而当前版本 ceph-deploy 不支持用户指定小版本号，使用--release只支持安装最新稳定版。当然我们可以手动搭建，但是操作下来耗时且容易出错，而且人生苦短（我太懒了）。

版本信息

  1. 系统版本

    [root@admin-node my-cluster]# cat /etc/system-release
    CentOS Linux release 7.6.1810 (Core) 
    [root@admin-node my-cluster]# uname -a
    Linux admin-node 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  2. ceph 版本

    [root@admin-node my-cluster]# ceph -v
    ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)

在本地配置离线存储库

安装网络服务

  1. 安装 ngix

    yum install epel-release -y
    yum install nginx -y

  2. 打开 nginx

    systemctl start nginx
    systemctl enable nginx
    systemctl status nginx

  3. 添加防火墙规则

    firewall-cmd --zone=public --permanent --add-service=http
    firewall-cmd --zone=public --permanent --add-service=https
    firewall-cmd --reload

创建本地镜像源

  1. 安装必需工具

    yum install createrepo yum-utils wget -y

  2. 创建镜像源文件夹

    mkdir -p /var/www/html/repos/{SRPMS,x86_64,noarch}

  3. 下载相应软件包
    可以手动在以下 3 个链接中下载各自架构下的相应软件包

    https://download.ceph.com/rpm-nautilus/el7/SRPMS/
    https://download.ceph.com/rpm-nautilus/el7/noarch/
    https://download.ceph.com/rpm-nautilus/el7/x86_64/

另外，我们也可以从阿里云的镜像源链接http://mirrors.aliyun.com/ceph/下载（国内用户你懂的）

wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-base-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-common-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-debuginfo-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-fuse-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mds-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mgr-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mon-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-osd-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-radosgw-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-resource-agents-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-selinux-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-test-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/cephfs-java-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs2-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs_jni-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs_jni1-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librados-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librados2-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libradosstriper-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libradosstriper1-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librbd-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librbd1-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librgw-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librgw2-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-ceph-compat-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-cephfs-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rados-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rbd-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rgw-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-ceph-argparse-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-cephfs-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rados-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rbd-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rgw-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rados-objclass-devel-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-fuse-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-mirror-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-nbd-14.2.5-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-ceph-argparse-14.2.5-0.el7.x86_64.rpm
wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ceph-deploy-2.0.0-0.noarch.rpm

  1. 执行镜像源创建

    createrepo -v /var/www/html/repos/SRPMS/
    createrepo -v /var/www/html/repos/x86_64/
    createrepo -v /var/www/html/repos/noarch/

  2. 修改 Nginx 配置

    vi /etc/nginx/conf.d/repos.conf
server {
        listen   80;
        server_name  172.18.1.101;	# 修改此处为本机地址
        root   /var/www/html/repos;
        location / {
                index  index.php index.html index.htm;
                autoindex on;	#enable listing of directory index
        }
}
  1. 重新启动 nginx，然后再次访问本地源测试
    systemctl restart nginx

upload successful

安装 Ceph

  1. 声明本地仓库
    vi/etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph packages for $basearch
baseurl=http://172.18.1.101/x86_64/		# 注意此处ip为镜像源的ip
enabled=1
priority=2
gpgcheck=0
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=http://172.18.1.101/noarch
enabled=1
priority=2
gpgcheck=0
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=http://172.18.1.101/SRPMS
enabled=0
priority=2
gpgcheck=0
gpgkey=https://download.ceph.com/keys/release.asc
EOF

  1. 然后像往常一样进行 ceph 部署

    yum -y install ceph-deploy

  2. 转到使用 ceph deploy 安装的步骤，我们将使用 –repo-url 替换命令

    ceph-deploy install --repo-url http://172.18.1.101/ ceph1 ceph2 ceph3

  3. 安装完成后，再次检查ceph version版本

    [node2][DEBUG ] 
    [node2][DEBUG ] Complete!
    [node2][INFO  ] Running command: sudo ceph --version
    [node2][DEBUG ] ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)
    [root@admin-node x86_64]# ceph -v
    ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)

总结

ceph-deploy 可以通过三种指定方式部署 ceph,分别为ceph-deploy install --release（用于指定大版本号如nautilus）、ceph-deploy install --testing(用于最新开发版本)和ceph-deploy install --dev（用于指定分支或者 tag）,起初我以为--dev可以满足我指定版本号的需求，后来一顿操作发现报错:

[ceph_deploy.hosts.centos.install][DEBUG ] fetching repo information from: https://shaman.ceph.com/api/repos/ceph/14.5.2/latest/centos/7/repo/?arch=x86_64
[ceph_deploy.util.net][ERROR ] repository might not be available yet
[ceph_deploy][ERROR ] RuntimeError: HTTP Error 504: Gateway Timeout, failed to fetch https://shaman.ceph.com/api/repos/ceph/14.5.2/latest/centos/7/repo/?arch=x86_64

访问https://shaman.ceph.com/api/repos/ceph发现可用库里根本没有对应版本。通过阅读install — ceph-deploy 2.0.2 documentation我们知道可以指定 --repo-url搭配--gpg-url使用指定源链接的方式指定本地源安装。此外，使用--local-mirrors指定本地源的方式也可以参考。参见install — ceph-deploy 2.0.2 documentation。

本文通过搭建本地镜像源然后指定的方式间接完成了 ceph 集群的安装。

参考链接