操作系统:CentOS Linux release 7.2.1511 (Core) 内核版本:3.10.0-327.28.3.el7.x86_64 kolla版本:mitaka-eol docker版本:Docker version 1.13.1, build 092cba3 docker镜像:官方tag 2.0.2 (对应 openstack mitaka版本)
问题一: openvswitch_db 容器无法运行
问题描述
kolla-ansible deploy 部署openstack的时候总会遇到 openvswitch_db service 无法启动的问题
TASK: [neutron | Waiting the openvswitch_db service to be ready] ************** failed: [localhost] => {"attempts": 30, "changed": false, "cmd": ["docker", "exec", "openvswitch_db", "ovs-vsctl", "--no-wait", "show"], "delta": "0:00:00.032518", "end": "2018-07-09 07:33:12.680647", "failed": true, "rc": 1, "start": "2018-07-09 07:33:12.648129", "stdout_lines": [], "warnings": []} stderr: Error response from daemon: Container 0cec739aabe06805aa0e1624318ac052d9f8fb176078df3d20a13c4df304fa7a is restarting, wait until the container is running msg: Task failed as maximum retries was encountered
FATAL: all hosts have already failed -- aborting
查看容器日志有如下报错
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:Writing out command to execute Running command: '/usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --log-file=/var/log/openvswitch/ovsdb-server.log' ovsdb-server: I/O error: open: /var/lib/openvswitch/conf.db failed (No such file or directory)
排查问题
1. 手动创建 openvswitch_db 容器,并进入交互模式
docker run -it kolla/centos-source-openvswitch-db-server:2.0.2 /bin/bash
2. 查看启动命令
# cd /usr/local/bin # vi kolla_start
#!/bin/bash set -o errexit
# Wait for the log socket if [[ ! "${!SKIP_LOG_SETUP[@]}" && -e /var/lib/kolla/heka ]]; then while [[ ! -S /var/lib/kolla/heka/log ]]; do sleep 1 done fi
# Processing /var/lib/kolla/config_files/config.json as root. This is necessary # to permit certain files to be controlled by the root user which should # not be writable by the dropped-privileged user, especially /run_command sudo -E kolla_set_configs CMD=$(cat /run_command) ARGS=""
if [[ ! "${!KOLLA_SKIP_EXTEND_START[@]}" ]]; then # Run additional commands if present source kolla_extend_start fi
REPOSITORY TAG IMAGE ID CREATED SIZE kolla/centos-binary-openvswitch-db-server 2.0.458e8a1cdc38710 minutes ago 379 MB kolla/centos-binary-openvswitch-vswitchd 2.0.4 cb85d198f02c 10 minutes ago 379 MB kolla/centos-binary-openvswitch-base 2.0.40baee06b57a410 minutes ago 379 MB kolla/centos-binary-kolla-toolbox 2.0.4 d3f9e86e129243 minutes ago 631 MB kolla/centos-binary-base 2.0.44481fe643afa About an hour ago 344 MB kolla/centos-binary-openvswitch-base 2.0.409ab40d1a6846 hours ago 380 MB kolla/centos-binary-ironic-inspector 2.0.40a692e9b679e 10 hours ago 641 MB kolla/centos-binary-ironic-api 2.0.4087a20a24a8410 hours ago 635 MB kolla/centos-binary-ironic-conductor 2.0.435450cd6b73b 10 hours ago 662 MB kolla/centos-binary-nova-compute-ironic 2.0.4 d0a5a0a85ab710 hours ago 1.07 GB kolla/centos-binary-ironic-pxe 2.0.4 d505622b698210 hours ago 639 MB kolla/centos-binary-elasticsearch 2.0.463064a9d79d110 hours ago 692 MB kolla/centos-binary-kibana 2.0.4 d52426e4d06f 10 hours ago 874 MB kolla/centos-binary-ironic-base 2.0.409a4902a06be 10 hours ago 612 MB kolla/centos-binary-nova-libvirt 2.0.481f523f3d65611 hours ago 1.11 GB kolla/centos-binary-nova-compute 2.0.45b70975fe56d 11 hours ago 1.11 GB kolla/centos-binary-cinder-volume 2.0.4 acc66141a64011 hours ago 859 MB kolla/centos-binary-cinder-api 2.0.45ff44bfe406311 hours ago 850 MB kolla/centos-binary-cinder-rpcbind 2.0.4 d241518b407c 11 hours ago 838 MB kolla/centos-binary-cinder-backup 2.0.470e279b6adc711 hours ago 808 MB kolla/centos-binary-cinder-scheduler 2.0.4 d4c8e5140be411 hours ago 808 MB kolla/centos-binary-glance-api 2.0.492ed32bb634411 hours ago 732 MB kolla/centos-binary-nova-conductor 2.0.4 c7f752689dfc 11 hours ago 671 MB kolla/centos-binary-nova-consoleauth 2.0.45bb4f725b42d 11 hours ago 671 MB kolla/centos-binary-nova-scheduler 2.0.400ddedda23c411 hours ago 671 MB kolla/centos-binary-glance-registry 2.0.4 e59b0948281e 11 hours ago 732 MB kolla/centos-binary-nova-ssh 2.0.401c404afdc8b 11 hours ago 672 MB kolla/centos-binary-nova-api 2.0.4 d29b3451c04511 hours ago 671 MB kolla/centos-binary-nova-network 2.0.4 a5cead8aee0b 11 hours ago 672 MB kolla/centos-binary-neutron-openvswitch-agent 2.0.436226e14b4d711 hours ago 670 MB kolla/centos-binary-neutron-linuxbridge-agent 2.0.4 e9ad4cb7c6cc 11 hours ago 670 MB kolla/centos-binary-nova-novncproxy 2.0.43689a51a5db811 hours ago 672 MB kolla/centos-binary-nova-spicehtml5proxy 2.0.455fc9d8a62b511 hours ago 672 MB kolla/centos-binary-neutron-metadata-agent 2.0.445f0f090cf3811 hours ago 646 MB kolla/centos-binary-cinder-base 2.0.4 dd0c5b78af7b 11 hours ago 808 MB kolla/centos-binary-neutron-server 2.0.4171b7bab73ab 11 hours ago 646 MB kolla/centos-binary-heat-api 2.0.4 f334caf10d5a 11 hours ago 633 MB kolla/centos-binary-horizon 2.0.488ceecbc8cf811 hours ago 763 MB kolla/centos-binary-heat-engine 2.0.4 a5365146323511 hours ago 633 MB kolla/centos-binary-heat-api-cfn 2.0.47ec6cdd4b04b 11 hours ago 633 MB kolla/centos-binary-neutron-l3-agent 2.0.4 d4744d180b0911 hours ago 646 MB kolla/centos-binary-neutron-dhcp-agent 2.0.4 d4744d180b0911 hours ago 646 MB kolla/centos-binary-glance-base 2.0.4 f78fa9de5c7b 11 hours ago 732 MB kolla/centos-binary-nova-base 2.0.4 aa7ae5ae481811 hours ago 648 MB kolla/centos-binary-neutron-base 2.0.4 ed5f4f60a6f411 hours ago 646 MB kolla/centos-binary-heat-base 2.0.44798734eb0d411 hours ago 610 MB kolla/centos-binary-keystone 2.0.455cf2686b33a 11 hours ago 644 MB kolla/centos-binary-openstack-base 2.0.49d511be689b722 hours ago 572 MB kolla/centos-binary-mariadb 2.0.4 cb4c65a6a63722 hours ago 682 MB kolla/centos-binary-openvswitch-vswitchd 2.0.40179076733aa 22 hours ago 380 MB kolla/centos-binary-openvswitch-db-server 2.0.4 a9e0e1bd096822 hours ago 380 MB kolla/centos-binary-rabbitmq 2.0.4 cbaeb0b6493022 hours ago 438 MB kolla/centos-binary-memcached 2.0.411aa506130a622 hours ago 403 MB kolla/centos-binary-heka 2.0.489c723045d4022 hours ago 420 MB kolla/centos-binary-cron 2.0.44b0b36b058a122 hours ago 366 MB kolla/centos-binary-keepalived 2.0.47fbc06505ddb 23 hours ago 411 MB kolla/centos-binary-haproxy 2.0.47f65eba4390923 hours ago 367 MB centos latest 49f7960eb7e45 weeks ago 200 MB
问题四: openvswitch_db 容器无法启动
问题描述
重新编译了镜像,可是 deploy 的时候又出现了以下错误
TASK: [neutron | Waiting the openvswitch_db service to be ready] ************** failed: [localhost] => {"attempts": 30, "changed": false, "cmd": ["docker", "exec", "openvswitch_db", "ovs-vsctl", "--no-wait", "show"], "delta": "0:00:00.057827", "end": "2018-07-12 08:04:09.663118", "failed": true, "rc": 1, "start": "2018-07-12 08:04:09.605291", "stdout_lines": [], "warnings": []} stderr: Error response from daemon: Container 03426314b560db08c762a8f9aebdb4423571a29ba1c22862e3415ac913289c21 is restarting, wait until the container is running msg: Task failed as maximum retries was encountered
FATAL: all hosts have already failed -- aborting
和问题一的报错一致,查看容器日志
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:Writing out command to execute Running command: '/usr/sbin/ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --log-file=/var/log/openvswitch/ovsdb-server.log' ovsdb-server: I/O error: open: /etc/openvswitch/conf.db failed (No such file or directory)
648c226f0980 kolla/centos-binary-nova-compute:2.0.4"kolla_start"3 days ago Restarting (0) 21 hours ago nova_compute c492de413c81 kolla/centos-binary-nova-libvirt:2.0.4"kolla_start"3 days ago Restarting (6) 21 hours ago nova_libvirt
查看对应容器的日志 nova_compute 容器
# docker logs 648c226f0980 INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:Removing existing destination: /etc/nova/nova.conf INFO:__main__:Copying /var/lib/kolla/config_files/nova.conf to /etc/nova/nova.conf INFO:__main__:Setting permissions for/etc/nova/nova.conf INFO:__main__:Writing out command to execute Running command: 'nova-compute' /usr/lib/python2.7/site-packages/pkg_resources/__init__.py:187: RuntimeWarning: You have iterated over the result of pkg_resources.parse_version. This is a legacy behavior which is inconsistent with the new version class introduced in setuptools 8.0. In most cases, conversion to a tuple is unnecessary. For comparison of versions, sort the Version instances directly. If you have another use case requiring the tuple, please file a bug with the setuptools project describing that need. stacklevel=1, Traceback (most recent call last): File"/usr/lib/python2.7/site-packages/eventlet/queue.py", line 118, in switch self.greenlet.switch(value) File"/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main result = function(*args, **kwargs) File"/usr/lib/python2.7/site-packages/oslo_service/service.py", line 683, in run_service raise SystemExit(1) SystemExit: 1
nova_libvirt 容器
# docker logs c492de413c81
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:Removing existing destination: /etc/libvirt/libvirtd.conf INFO:__main__:Copying /var/lib/kolla/config_files/libvirtd.conf to /etc/libvirt/libvirtd.conf INFO:__main__:Setting permissions for/etc/libvirt/libvirtd.conf INFO:__main__:Removing existing destination: /etc/libvirt/qemu.conf INFO:__main__:Copying /var/lib/kolla/config_files/qemu.conf to /etc/libvirt/qemu.conf INFO:__main__:Setting permissions for/etc/libvirt/qemu.conf INFO:__main__:Writing out command to execute Running command: '/usr/sbin/libvirtd --listen'
cffc9fc2774b kolla/centos-binary-nova-compute:2.0.4"kolla_start"5 minutes ago Up 5 minutes nova_compute 5f27c8052238 kolla/centos-binary-nova-libvirt:2.0.4"kolla_start"5 minutes ago Up 5 minutes nova_libvirt