之前用的是 ceph-deploy 部署 ceph 集群,在官网的最新介绍中有如下描述:
ceph-deploy is no longer actively maintained. It is not tested on versions of Ceph newer than Nautilus. It does not support RHEL8, CentOS 8, or newer operating systems.
ceph-deploy 已经不在维护,并且在 ceph Nautilus 之后都没有很好的测试,不支持 RHEL8、CentOS8等系统。对比了 ceph-ansible 和 cephadm 这两个工具,最终选择的 ceph-ansible 作为部署工具。
主机角色
Hostname
Role
Admin/Public Network
Cluster Network
OS
node01.ceph.local
ceph-ansible,mon,mgr,osd,rgw,mds,grafana
192.168.198.131/24
172.20.1.131/24
CentOS Linux release 8.3.2011
node02.ceph.local
mon,mgr,osd,rgw,mds
192.168.198.132/24
172.20.1.132/24
CentOS Linux release 8.3.2011
node03.ceph.local
mon,mgr,osd,rgw,mds
192.168.198.133/24
172.20.1.133/24
CentOS Linux release 8.3.2011
前期准备 包括配置IP
、配置主机名
、配置HOSTS文件
、配置加速源(EPEL)
、配置时钟同步
、关闭 SELinux
、关闭Firewalld
、设置免密登录
等,不再赘述。
Ansible配置 1.ansible安装 使用以下命令安装 ansible, ceph-ansible stable 5.x 需要 ansible 2.9 源中的版本满足需求,可以直接 yum 安装
2.主机配置 编辑 /etc/ansible/hosts 加入以下内容
[mons] node01.ceph.local node02.ceph.local node03.ceph.local [mgrs] node01.ceph.local node02.ceph.local node03.ceph.local [osds] node01.ceph.local node02.ceph.local node03.ceph.local [rgws] node01.ceph.local node02.ceph.local node03.ceph.local [mdss] node01.ceph.local node02.ceph.local node03.ceph.local [grafana-server] node01.ceph.local
ceph-ansible配置 1.下载 ceph-ansible 下载 5.x 版本的 ceph-ansible
2.安装 ceph-ansible 依赖 # tar zxvf v5.0.3.tar.gz # yum install python-pip # pip install -r ceph-ansible/requirements.txt
3.修改安装 ceph 配置 --- dummy: ceph_repository_type: repository ceph_origin: repository ceph_repository: community ceph_mirror: https://mirrors.tuna.tsinghua.edu.cn/ceph/ ceph_stable_key: https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc ceph_stable_release: octopus ceph_stable_repo: https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.7/el7/ monitor_interface: eth0 public_network: 192.168 .198 .0 /24 cluster_network: 172.20 .1 .0 /24 osd_objectstore: bluestore radosgw_civetweb_port: 8080 radosgw_interface: eth0 dashboard_enabled: True dashboard_admin_user: admin dashboard_admin_password: p@ssw0rd grafana_admin_user: admin grafana_admin_password: p@ssw0rd ceph_conf_overrides: global: mon_allow_pool_delete: true mon_osd_allow_primary_affinity: 1 mon_clock_drift_allowed: 0.5 osd_pool_default_size: 3 osd_pool_default_min_size: 1 mon_pg_warn_min_per_osd: 0 mon_pg_warn_max_per_osd: 0 mon_pg_warn_max_object_skew: 0 client: rbd_default_features: 1
4.osd配置 按照服务器配置的磁盘修改以下配置文件
--- dummy: devices: - /dev/sdb - /dev/sdc
5.其他配置 复制其他的配置文件
# cp clients.yml.sample clients.yml # cp mons.yml.sample mons.yml # cp mgrs.yml.sample mgrs.yml # cp rgws.yml.sample rgws.yml # cd ../ # cp site.yml.sample site.yml
开始安装 执行以下命令开始安装
# ansible-playbook site.yml
完成安装后查看集群状态:
cluster : id : 6 e344 dd2 -341 a-4 bb6 -aafa-4299 a0 ebbe51 health : HEALTH_OK services : mon : 3 daemons, quorum node01 .ceph.local,node02 .ceph.local,node03 .ceph.local (age 6 m) mgr : node01 .ceph.local(active, since 40 s), standbys: node02 .ceph.local, node02 .ceph.local mds : cephfs:1 {0 =node01 .ceph.local=up:active} 2 up:standby osd : 12 osds: 12 up (since 2 m), 12 in (since 23 m) rgw : 3 daemons active (node01 .ceph.local.rgw0 , node02 .ceph.local.rgw0 , 3 .ceph.local.rgw0 ) data : pools : 6 pools, 144 pgs objects : 212 objects, 6 .4 KiB usage : 12 GiB used, 9 .8 TiB / 9 .8 TiB avail pgs : 144 active+clean
清理安装 如果安装出错,或者遇到其他问题,可以清理集群后再次尝试安装
# ansible-playbook infrastructure-playbooks/purge-cluster.yml