实验环境

主机名	IP
controller1	192.168.2.240
compute1	192.168.2.242
compute2	192.168.2.243
compute3	192.168.2.248
compute4	192.168.2.249

在不同的计算节点使用不同的存储后端

计算节点配置

1. Scheduler

为了使nova的调度程序支持下面的过滤算法，需要修改使之支持 AggregateInstanceExtraSpecsFilter ，编辑控制节点的 /etc/nova/nova.conf 文件加入修改以下选项,然后重启nova-scheduler服务

1
2
3
4


# vim /etc/nova/nova.conf
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateInstanceExtraSpecsFilter

# systemctl restart openstack-nova-scheduler.service

2. 本地存储配置

nova默认支持，无需配置。为了支持迁移可以配置共享存储（NFS等）

3. ceph存储配置

编辑计算节点的 /etc/nova/nova.conf 文件加入修改以下选项,然后重启nova-compute服务（这里没有详细写，例如导入secret-uuid等操作请自行添加）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# vim /etc/nova/nova.conf
[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid =20c3fd98-2bab-457a-b1e2-12e50dc6c98e
disk_cachemodes="network=writeback"
inject_partition=-2
inject_key=False
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST

# systemctl restart openstack-nova-compute.service

openstack配置

创建主机集合，包含ceph compute nodes 和 local storage compute nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# nova aggregate-create ephemeral-compute-storage
+----+---------------------------+-------------------+-------+----------+
| Id | Name                      | Availability Zone | Hosts | Metadata |
+----+---------------------------+-------------------+-------+----------+
| 8  | ephemeral-compute-storage | -                 |       |          |
+----+---------------------------+-------------------+-------+----------+

# nova aggregate-create ceph-compute-storage
+----+----------------------+-------------------+-------+----------+
| Id | Name                 | Availability Zone | Hosts | Metadata |
+----+----------------------+-------------------+-------+----------+
| 9  | ceph-compute-storage | -                 |       |          |
+----+----------------------+-------------------+-------+----------+

可以使用 nova hypervisor-list 命令来查看自己的hypervisor name

1
2
3
4
5
6
7
8
9


# nova hypervisor-list
+----+---------------------+-------+---------+
| ID | Hypervisor hostname | State | Status  |
+----+---------------------+-------+---------+
| 1  | compute1            | up    | enabled |
| 2  | compute2            | up    | enabled |
| 4  | compute4            | up    | enabled |
| 7  | compute3            | up    | enabled |
+----+---------------------+-------+---------+

在本例中，使用以下分类 Local storage:compute1,compute2 Ceph storage: conpute3,compute4 添加主机到主机集合

1
2
3
4


# nova aggregate-add-host ephemeral-compute-storage compute1
# nova aggregate-add-host ephemeral-compute-storage compute2
# nova aggregate-add-host ceph-compute-storage compute3
# nova aggregate-add-host ceph-compute-storage compute4

为主机集合创建新的metadata

1
2


# nova aggregate-set-metadata ephemeral-compute-storage ephemeralcomputestorage=true
# nova aggregate-set-metadata ceph-compute-storage cephcomputestorage=true

为使用本地存储和ceph存储的虚拟机创建flavor

1
2


# nova flavor-create m1.ephemeral-compute-storage 8 128 1 1
# nova flavor-create m1.ceph-compute-storage 9 128 1 1

为flavor绑定指定的属性

1
2


# nova flavor-key m1.ceph-compute-storage set aggregate_instance_extra_specs:cephcomputestorage=true
# nova flavor-key m1.ephemeral-compute-storage set aggregate_instance_extra_specs:ephemeralcomputestorage=true

结果验证

使用flavor m1.ceph-compute-storage 启动4台虚拟机，发现虚拟机磁盘文件全部在ceph的pool中

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID                                   | Name   | Status | Task State | Power State | Networks            |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 5d6bd85e-9b75-4035-876c-30e997ea0a98 | ceph-1 | BUILD  | spawning   | NOSTATE     | private=172.16.1.49 |
| aa666bd9-e370-4c53-8af3-f1bf7ba77900 | ceph-2 | BUILD  | spawning   | NOSTATE     | private=172.16.1.48 |
| 56d6a3a8-e6c4-4860-bd72-2e0aa0fa55f2 | ceph-3 | BUILD  | spawning   | NOSTATE     | private=172.16.1.47 |
| 2b9577d8-2448-4d8a-ba98-253b0f597b12 | ceph-4 | BUILD  | spawning   | NOSTATE     | private=172.16.1.50 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+

[root@node1 ~]# rbd ls vms
2b9577d8-2448-4d8a-ba98-253b0f597b12_disk
56d6a3a8-e6c4-4860-bd72-2e0aa0fa55f2_disk
5d6bd85e-9b75-4035-876c-30e997ea0a98_disk
aa666bd9-e370-4c53-8af3-f1bf7ba77900_disk

删除所有虚拟机（便于验证），使用flavor m1.ephemeral-compute-storage 启动四台虚拟机，发现虚拟机磁盘文件分布于compute1 和 compute2 的本地存储中（没有配置NFS等共享存储）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks            |
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| 1c1ce5f3-b5aa-47dd-806c-e2eba60b9eb0 | local-1 | ACTIVE | -          | Running     | private=172.16.1.51 |
| 5a3e4074-619e-423a-a649-e24771f9fbd1 | local-2 | ACTIVE | -          | Running     | private=172.16.1.54 |
| 5b838406-b9cf-4943-89f3-79866f8e6e19 | local-3 | ACTIVE | -          | Running     | private=172.16.1.52 |
| 30e7289f-bc80-4374-aabb-906897b8141c | local-4 | ACTIVE | -          | Running     | private=172.16.1.53 |
+--------------------------------------+---------+--------+------------+-------------+---------------------+

[root@compute1 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x 2 nova nova  69 Jul 27 10:40 1c1ce5f3-b5aa-47dd-806c-e2eba60b9eb0
drwxr-xr-x 2 nova nova  69 Jul 27 10:40 5b838406-b9cf-4943-89f3-79866f8e6e19
drwxr-xr-x 2 nova nova  53 Jul 25 16:01 _base
-rw-r--r-- 1 nova nova  31 Jul 27 10:33 compute_nodes
drwxr-xr-x 2 nova nova 143 Jul 25 16:01 locks
drwxr-xr-x 2 nova nova   6 Jul  6 15:51 snapshots

[root@compute2 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x 2 nova nova  69 Jul 27 10:40 30e7289f-bc80-4374-aabb-906897b8141c
drwxr-xr-x 2 nova nova  69 Jul 27 10:40 5a3e4074-619e-423a-a649-e24771f9fbd1
drwxr-xr-x 2 nova nova  53 Jul 25 16:02 _base
-rw-r--r-- 1 nova nova  62 Jul 27 10:33 compute_nodes
drwxr-xr-x 2 nova nova 143 Jul 25 16:01 locks