实验环境
| 主机名 |
IP |
| controller1 |
192.168.2.240 |
| compute1 |
192.168.2.242 |
| compute2 |
192.168.2.243 |
| compute3 |
192.168.2.248 |
| compute4 |
192.168.2.249 |
在不同的计算节点使用不同的存储后端
计算节点配置
1. Scheduler
为了使nova的调度程序支持下面的过滤算法，需要修改使之支持 AggregateInstanceExtraSpecsFilter ，编辑控制节点的 /etc/nova/nova.conf 文件加入修改以下选项,然后重启nova-scheduler服务
1
2
3
4
|
# vim /etc/nova/nova.conf
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateInstanceExtraSpecsFilter
# systemctl restart openstack-nova-scheduler.service
|
2. 本地存储配置
nova默认支持，无需配置。为了支持迁移可以配置共享存储（NFS等）
3. ceph存储配置
编辑计算节点的 /etc/nova/nova.conf 文件加入修改以下选项,然后重启nova-compute服务（这里没有详细写，例如导入secret-uuid等操作请自行添加）
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# vim /etc/nova/nova.conf
[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid =20c3fd98-2bab-457a-b1e2-12e50dc6c98e
disk_cachemodes="network=writeback"
inject_partition=-2
inject_key=False
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST
# systemctl restart openstack-nova-compute.service
|
openstack配置
创建主机集合，包含ceph compute nodes 和 local storage compute nodes
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# nova aggregate-create ephemeral-compute-storage
+----+---------------------------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------------------------+-------------------+-------+----------+
| 8 | ephemeral-compute-storage | - | | |
+----+---------------------------+-------------------+-------+----------+
# nova aggregate-create ceph-compute-storage
+----+----------------------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+----------------------+-------------------+-------+----------+
| 9 | ceph-compute-storage | - | | |
+----+----------------------+-------------------+-------+----------+
|
可以使用 nova hypervisor-list 命令来查看自己的hypervisor name
1
2
3
4
5
6
7
8
9
|
# nova hypervisor-list
+----+---------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+----+---------------------+-------+---------+
| 1 | compute1 | up | enabled |
| 2 | compute2 | up | enabled |
| 4 | compute4 | up | enabled |
| 7 | compute3 | up | enabled |
+----+---------------------+-------+---------+
|
在本例中，使用以下分类
Local storage:compute1,compute2
Ceph storage: conpute3,compute4
添加主机到主机集合
1
2
3
4
|
# nova aggregate-add-host ephemeral-compute-storage compute1
# nova aggregate-add-host ephemeral-compute-storage compute2
# nova aggregate-add-host ceph-compute-storage compute3
# nova aggregate-add-host ceph-compute-storage compute4
|
为主机集合创建新的metadata
1
2
|
# nova aggregate-set-metadata ephemeral-compute-storage ephemeralcomputestorage=true
# nova aggregate-set-metadata ceph-compute-storage cephcomputestorage=true
|
为使用本地存储和ceph存储的虚拟机创建flavor
1
2
|
# nova flavor-create m1.ephemeral-compute-storage 8 128 1 1
# nova flavor-create m1.ceph-compute-storage 9 128 1 1
|
为flavor绑定指定的属性
1
2
|
# nova flavor-key m1.ceph-compute-storage set aggregate_instance_extra_specs:cephcomputestorage=true
# nova flavor-key m1.ephemeral-compute-storage set aggregate_instance_extra_specs:ephemeralcomputestorage=true
|
结果验证
使用flavor m1.ceph-compute-storage 启动4台虚拟机，发现虚拟机磁盘文件全部在ceph的pool中
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
# nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 5d6bd85e-9b75-4035-876c-30e997ea0a98 | ceph-1 | BUILD | spawning | NOSTATE | private=172.16.1.49 |
| aa666bd9-e370-4c53-8af3-f1bf7ba77900 | ceph-2 | BUILD | spawning | NOSTATE | private=172.16.1.48 |
| 56d6a3a8-e6c4-4860-bd72-2e0aa0fa55f2 | ceph-3 | BUILD | spawning | NOSTATE | private=172.16.1.47 |
| 2b9577d8-2448-4d8a-ba98-253b0f597b12 | ceph-4 | BUILD | spawning | NOSTATE | private=172.16.1.50 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
[root@node1 ~]# rbd ls vms
2b9577d8-2448-4d8a-ba98-253b0f597b12_disk
56d6a3a8-e6c4-4860-bd72-2e0aa0fa55f2_disk
5d6bd85e-9b75-4035-876c-30e997ea0a98_disk
aa666bd9-e370-4c53-8af3-f1bf7ba77900_disk
|
删除所有虚拟机（便于验证），使用flavor m1.ephemeral-compute-storage 启动四台虚拟机，发现虚拟机磁盘文件分布于compute1 和 compute2 的本地存储中（没有配置NFS等共享存储）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| 1c1ce5f3-b5aa-47dd-806c-e2eba60b9eb0 | local-1 | ACTIVE | - | Running | private=172.16.1.51 |
| 5a3e4074-619e-423a-a649-e24771f9fbd1 | local-2 | ACTIVE | - | Running | private=172.16.1.54 |
| 5b838406-b9cf-4943-89f3-79866f8e6e19 | local-3 | ACTIVE | - | Running | private=172.16.1.52 |
| 30e7289f-bc80-4374-aabb-906897b8141c | local-4 | ACTIVE | - | Running | private=172.16.1.53 |
+--------------------------------------+---------+--------+------------+-------------+---------------------+
[root@compute1 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x 2 nova nova 69 Jul 27 10:40 1c1ce5f3-b5aa-47dd-806c-e2eba60b9eb0
drwxr-xr-x 2 nova nova 69 Jul 27 10:40 5b838406-b9cf-4943-89f3-79866f8e6e19
drwxr-xr-x 2 nova nova 53 Jul 25 16:01 _base
-rw-r--r-- 1 nova nova 31 Jul 27 10:33 compute_nodes
drwxr-xr-x 2 nova nova 143 Jul 25 16:01 locks
drwxr-xr-x 2 nova nova 6 Jul 6 15:51 snapshots
[root@compute2 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x 2 nova nova 69 Jul 27 10:40 30e7289f-bc80-4374-aabb-906897b8141c
drwxr-xr-x 2 nova nova 69 Jul 27 10:40 5a3e4074-619e-423a-a649-e24771f9fbd1
drwxr-xr-x 2 nova nova 53 Jul 25 16:02 _base
-rw-r--r-- 1 nova nova 62 Jul 27 10:33 compute_nodes
drwxr-xr-x 2 nova nova 143 Jul 25 16:01 locks
|
补充说明
在线迁移虚拟机的时候，不在同一个主机集合的主机仍然可以选择，但是无法迁移，需要增加只能在所在主机集合内迁移的功能
参考文章
OpenStack: use ephemeral and persistent root storage for different hypervisors