使用ceph-ansible部署Ceph Octopus

之前用的是 ceph-deploy 部署 ceph 集群,在官网的最新介绍中有如下描述:

ceph-deploy is no longer actively maintained. It is not tested on versions of Ceph newer than Nautilus. It does not support RHEL8, CentOS 8, or newer operating systems.

ceph-deploy 已经不在维护,并且在 ceph Nautilus 之后都没有很好的测试,不支持 RHEL8、CentOS8等系统。对比了 ceph-ansible 和 cephadm 这两个工具,最终选择的 ceph-ansible 作为部署工具。

主机角色

Hostname Role Admin/Public Network Cluster Network OS
node01.ceph.local ceph-ansible,mon,mgr,osd,rgw,mds,grafana 192.168.198.131/24 172.20.1.131/24 CentOS Linux release 8.3.2011
node02.ceph.local mon,mgr,osd,rgw,mds 192.168.198.132/24 172.20.1.132/24 CentOS Linux release 8.3.2011
node03.ceph.local mon,mgr,osd,rgw,mds 192.168.198.133/24 172.20.1.133/24 CentOS Linux release 8.3.2011

前期准备

包括配置IP配置主机名配置HOSTS文件配置加速源(EPEL)配置时钟同步关闭 SELinux关闭Firewalld设置免密登录等,不再赘述。

Ansible配置

1.ansible安装

使用以下命令安装 ansible, ceph-ansible stable 5.x 需要 ansible 2.9 源中的版本满足需求,可以直接 yum 安装

1
# yum install ansible -y

2.主机配置

编辑 /etc/ansible/hosts 加入以下内容

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# vim /etc/ansible/hosts

[mons]
node01.ceph.local
node02.ceph.local
node03.ceph.local

[mgrs]
node01.ceph.local
node02.ceph.local
node03.ceph.local

[osds]
node01.ceph.local
node02.ceph.local
node03.ceph.local

[rgws]
node01.ceph.local
node02.ceph.local
node03.ceph.local

[mdss]
node01.ceph.local
node02.ceph.local
node03.ceph.local

[grafana-server]
node01.ceph.local

ceph-ansible配置

1.下载 ceph-ansible

下载 5.x 版本的 ceph-ansible

1
# wget https://github.com/ceph/ceph-ansible/archive/v5.0.3.tar.gz

2.安装 ceph-ansible 依赖

1
2
3
# tar zxvf v5.0.3.tar.gz
# yum install python-pip
# pip install -r ceph-ansible/requirements.txt

3.修改安装 ceph 配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# cd ceph-ansible/group_vars/
# mv all.yml.sample all.yml
# grep -Ev '^#|^$' all.yml
---
dummy:
ceph_repository_type: repository
ceph_origin: repository
ceph_repository: community
ceph_mirror: https://mirrors.tuna.tsinghua.edu.cn/ceph/
ceph_stable_key: https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc
ceph_stable_release: octopus
ceph_stable_repo: https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.7/el7/
monitor_interface: eth0
public_network: 192.168.198.0/24
cluster_network: 172.20.1.0/24
osd_objectstore: bluestore
radosgw_civetweb_port: 8080
radosgw_interface: eth0
dashboard_enabled: True
dashboard_admin_user: admin
dashboard_admin_password: p@ssw0rd
grafana_admin_user: admin
grafana_admin_password: p@ssw0rd
ceph_conf_overrides:
    global:
        mon_allow_pool_delete: true
        mon_osd_allow_primary_affinity: 1
        mon_clock_drift_allowed: 0.5
        osd_pool_default_size: 3
        osd_pool_default_min_size: 1
        mon_pg_warn_min_per_osd: 0
        mon_pg_warn_max_per_osd: 0
        mon_pg_warn_max_object_skew: 0
    client:
        rbd_default_features: 1

4.osd配置

按照服务器配置的磁盘修改以下配置文件

1
2
3
4
5
6
7
# cp osds.yml.sample osds.yml
# grep -Ev '^#|^$' osds.yml
---
dummy:
devices:
  - /dev/sdb
  - /dev/sdc

5.其他配置

复制其他的配置文件

1
2
3
4
5
6
7
# cp clients.yml.sample clients.yml
# cp mons.yml.sample mons.yml
# cp mgrs.yml.sample mgrs.yml
# cp rgws.yml.sample rgws.yml

# cd ../
# cp site.yml.sample site.yml

开始安装

执行以下命令开始安装

1
# ansible-playbook site.yml

完成安装后查看集群状态:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# ceph -s
  cluster:
    id:     6e344dd2-341a-4bb6-aafa-4299a0ebbe51
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node01.ceph.local,node02.ceph.local,node03.ceph.local (age 6m)
    mgr: node01.ceph.local(active, since 40s), standbys: node02.ceph.local, node02.ceph.local
    mds: cephfs:1 {0=node01.ceph.local=up:active} 2 up:standby
    osd: 12 osds: 12 up (since 2m), 12 in (since 23m)
    rgw: 3 daemons active (node01.ceph.local.rgw0, node02.ceph.local.rgw0, 3.ceph.local.rgw0)

  data:
    pools:   6 pools, 144 pgs
    objects: 212 objects, 6.4 KiB
    usage:   12 GiB used, 9.8 TiB / 9.8 TiB avail
    pgs:     144 active+clean

清理安装

如果安装出错,或者遇到其他问题,可以清理集群后再次尝试安装

1
# ansible-playbook infrastructure-playbooks/purge-cluster.yml
Licensed under CC BY-NC-SA 4.0
Nickname
Email
Website
0/500
  • OωO
  • |´・ω・)ノ
  • ヾ(≧∇≦*)ゝ
  • (☆ω☆)
  • (╯‵□′)╯︵┴─┴
  •  ̄﹃ ̄
  • (/ω\)
  • ∠( ᐛ 」∠)_
  • (๑•̀ㅁ•́ฅ)
  • →_→
  • ୧(๑•̀⌄•́๑)૭
  • ٩(ˊᗜˋ*)و
  • (ノ°ο°)ノ
  • (´இ皿இ`)
  • ⌇●﹏●⌇
  • (ฅ´ω`ฅ)
  • (╯°A°)╯︵○○○
  • φ( ̄∇ ̄o)
  • ヾ(´・ ・`。)ノ"
  • ( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
  • (ó﹏ò。)
  • Σ(っ °Д °;)っ
  • ( ,,´・ω・)ノ"(´っω・`。)
  • ╮(╯▽╰)╭
  • o(*////▽////*)q
  • >﹏<
  • ( ๑´•ω•) "(ㆆᴗㆆ)
  • 😂
  • 😀
  • 😅
  • 😊
  • 🙂
  • 🙃
  • 😌
  • 😍
  • 😘
  • 😜
  • 😝
  • 😏
  • 😒
  • 🙄
  • 😳
  • 😡
  • 😔
  • 😫
  • 😱
  • 😭
  • 💩
  • 👻
  • 🙌
  • 🖕
  • 👍
  • 👫
  • 👬
  • 👭
  • 🌚
  • 🌝
  • 🙈
  • 💊
  • 😶
  • 🙏
  • 🍦
  • 🍉
  • 😣
  • 颜文字
  • Emoji
  • Bilibili
2 comments
kirito

waline 评论测试

 加利福尼亚
 Windows 10
 Microsoft Edge 92.0.902.62
Anonymous

hello..你用的系统是CentOS8,但是在修改安装 ceph 配置,ceph_stable_repo: https://mirrors.tuna.tsinghua.edu.cn/post/rpm-15.2.7/el7/这里用的却是el7?

 香港
 Windows 10
 Chrome 88.0.4324.96
tanmx
Reply @Anonymous :

@Anonymous , 是8 这里直接复制之前的 没注意 感谢提醒

 台湾
 macOS Big Sur
 Chrome 88.0.4324.96
一个默默无闻的工程师的日常
Built with Hugo
主题 StackJimmy 设计