发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：
# encoding=utf-8

import requests
from bs4 import BeautifulSoup
import csv
import codecs

DOWNLOAD_URL = 'http://movie.douban.com/top250'


def download_page(url):
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'}
    data = requests.get(url, headers=headers).content
    return data


def parse_html(html):
    #更换解析器，详情 https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#id9
    soup = BeautifulSoup(html, "lxml")
    movies_list_soup = soup.find('ol', attrs={'class': 'grid_view'})
    movie_name_list = []
    movie_rating_num_list = []
    movie_quote_list = []

    for movies_li in movies_list_soup.find_all('li'):
        movie_name = movies_li.find('div', attrs={'class': 'hd'}).find('span', attrs={'class': 'title'}).getText()
        movie_rating = movies_li.find('div', attrs={'class': 'bd'}).find('span', attrs={'class': 'rating_num',
                                                                                        'property': 'v:average'}).getText()
        #movie_quote = movies_li.find('div', attrs={'class': 'bd'}).find('span', attrs={'class': 'inq'}).getText()
        movie_quote_obj = movies_li.find('div', attrs={'class': 'bd'}).find('span', attrs={'class': 'inq'})
        ### 发现某部电影评价为空，此处判断评价是否为空
        if movie_quote_obj is None:
            movie_quote = ''
        else:
            movie_quote = movie_quote_obj.getText()

        movie_name_list.append(movie_name)
        movie_rating_num_list.append(movie_rating)
        movie_quote_list.append(movie_quote)

    next_page = soup.find('span', attrs={'class': 'next'}).find('a')
    if next_page:
        return movie_name_list, movie_rating_num_list, movie_quote_list, DOWNLOAD_URL + next_page['href']

    return movie_name_list, movie_rating_num_list, movie_quote_list, None


def main():
    url = DOWNLOAD_URL
    #utf_8_sig 解决windows下csv文件乱码问题
    with codecs.open('movies_top_250.csv', 'wb', encoding='utf_8_sig') as csvFile:
        writer = csv.writer(csvFile)
        writer.writerow((u'电影名', u'评分', u'理由'))
        while url:
            html = download_page(url)
            movies, rating, quote, url = parse_html(html)
            for i in range(len(movies)):
                writer.writerow((movies[i], rating[i], quote[i]))
                
if __name__ == '__main__':
    main()

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

安装Gnocchi

先决条件

1. 完成以下步骤，创建数据库

  • 使用 root 用户登录数据库

    $ mysql -u root -p

  • 创建 gnocchi 数据库

    > CREATE DATABASE gnocchi;

  • 配置 gnocchi 数据库权限（这里密码为 GNOCCHI_DBPASS，按需修改）

    > GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'localhost' IDENTIFIED BY 'GNOCCHI_DBPASS';
    > GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'%' IDENTIFIED BY 'GNOCCHI_DBPASS';

  • 退出数据库

2. 加载 admin 用户认证文件

$ . admin-openrc

3. 完成以下步骤创建服务认证

  • 创建 gnocchi 用户（用户密码为 GNOCCHI_PASS）

    $ openstack user create --domain default --password-prompt gnocchi

  • 为 gnocchi 用户增加 admin 的角色

    $ openstack role add --project service --user gnocchi admin

  • 创建 gnocchi 服务实体

    $ openstack service create --name gnocchi --description "Metric Service" metric
  1. 创建 gnocchi 服务API endpoints
    $ openstack endpoint create --region RegionOne metric public http://controller:8041
    $ openstack endpoint create --region RegionOne metric internal http://controller:8041
    $ openstack endpoint create --region RegionOne metric admin http://controller:8041
阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

问题描述

原有的 rabbitmq 集群出现问题，无法启动，尝试删除 /var/lib/rabbitmq/.erlang.cookie 重新组集群，依旧无法启动

# systemctl start rabbitmq-server.service
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

解决分析

查看错误日志

# journalctl -xe
-- Subject: Unit rabbitmq-server.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rabbitmq-server.service has begun starting up.
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: BOOT FAILED
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: ===========
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: Error description:
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: {error,{inconsistent_cluster,"Node rabbit@controller03 thinks it's clustered with node rabbit@controller02, but rabbit@controller02 disagrees"}}
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: Log files (may contain more information):
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: /var/log/rabbitmq/rabbit@controller03.log
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: /var/log/rabbitmq/rabbit@controller03-sasl.log
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: Stack trace:
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: [{rabbit_mnesia,check_cluster_consistency,0,
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: [{file,"src/rabbit_mnesia.erl"},{line,598}]},
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: {rabbit,'-boot/0-fun-0-',0,[{file,"src/rabbit.erl"},{line,275}]},
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,296}]},
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: {init,start_it,1,[]},
Nov 24 14:26:20 controller03 rabbitmq-server[13522]: {init,start_em,1,[]}]
Nov 24 14:26:21 controller03 rabbitmq-server[13522]: {"init terminating in do_boot",{error,{inconsistent_cluster,"Node rabbit@controller03 thinks it's clustered with node rabbit@controller02, but rabbit@controller02 disagrees"}}}
Nov 24 14:26:21 controller03 rabbitmq-server[13522]: init terminating in do_boot ()
Nov 24 14:26:22 controller03 rabbitmq-server[13522]: Crash dump is being written to: erl_crash.dump...done
Nov 24 14:26:22 controller03 systemd[1]: rabbitmq-server.service: main process exited, code=exited, status=1/FAILURE

阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

线上的rabbitmq集群经常无法访问，集群状态也不正常，查找到如下资料：

一. 环境概述

1. 3个rabbitmq node组成的集群镜像，当客户端无法连接或者连接rabbitmq超时的时候，集群的状态如下：

# rabbitmqctl cluster_status
Cluster status of node rabbit@controller02 ...
[{nodes,[{disc,[rabbit@controller01]},
         {ram,[rabbit@controller03,rabbit@controller02]}]},
 {running_nodes,[rabbit@controller03,rabbit@controller02]},
 {cluster_name,<<"rabbit@controller01">>},
 {partitions,[{rabbit@controller02,[rabbit@controller01]}]},
 {alarms,[{rabbit@controller03,[]},{rabbit@controller02,[]}]}]

可以看到partitions中有内容，发生了网络分区。

阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

Rally 基本介绍

Rally 是OpenStack社区推出开源测试工具，可用于对OpenStack各个组件进行性能测试。通过使用Rally组件，用户可完成OpenStack云计算平台的安装部署、功能验证、大规模负载测试(性能测试)、输出测试报告等一系列动作。
Rally 的概况和结构如下图所示：
image
Rally 主要包括三大部分：

  • Deploy engine：这不是一个真的部署工具，它只是一个插件形式的东西，它可以和其他部署工具（比如 DevStack，Fuel，Anvil 等）一起工作来简化和统一部署流程。
  • Verification：使用tempest验证已经部署的openstack云环境的功能。
  • Benchmark engine：性能测试

Tempest 基本介绍

Tempest 是一个旨在为云计算平台 OpenStack 提供集成功能测试的开源项目，包含了 Openstack 基本组件（nova, keystone, glance, neutron, cinder 等）的 API 测试用例与场景。它是基于 unittest2 和 nose 建立的，灵活且易于扩展及维护，使得 OpenStack 相关测试效率得到大幅度提升。

安装 Rally

阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

系统环境

  • 至少需要三台虚拟机或者物理机，这里使用虚拟机
  • 每台虚拟机至少需要两块硬盘（一块系统盘，一块OSD），本例中有三块硬盘

1. 部署流程

博客使用的markdown解析器不支持流程图使用图片代替
liuchengtu.png

2. 主机规划

biaoge.png

安装 docker

登录 https://cr.console.aliyun.com/#/accelerator 获取自己的阿里云 docker 加速地址

1. 安装升级 docker 客户端

# curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -

2. 使用 docker 加速器

可以通过修改 daemon 配置文件 /etc/docker/daemon.json 来使用加速器，注意修改使用自己的加速地址

# mkdir -p /etc/docker
# tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://******.mirror.aliyuncs.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

在ocata的版本中，引入了一个新的“openstack profiler”的面板，启用openstack profiler可以方便的看到访问horizon页面时的API调用情况。如下图所示：
image
下面介绍如何启用 openstack profiler，首先需要一个正常运行的devstack环境，启用方法如下

安装mongoDB

Horizon会将API调用过程的数据都保存到mongodb中，mongodb可以安装在本机，也可以在本机能够访问的任意一台机器上。

1. 安装软件包

# yum install mongodb-server mongodb -y
阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

ssh 无密码登录失败

虚拟机 resize 需要配置计算节点之间 nova 用户无密码访问，但是在配置过程中有一台始终不能用密钥登录，对比了正常可以无密码登录的日志如下。

# 正常登录
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/nova/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279

阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

控制台无法访问，多次刷新才能访问，nova有如下报错

2017-02-09 17:09:51.311 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] "GET /websockify HTTP/1.1" 101 -
2017-02-09 17:09:51.312 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Plain non-SSL (ws://) WebSocket connection
2017-02-09 17:09:51.313 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Version hybi-13, base64: 'False'
2017-02-09 17:09:51.313 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Path: '/websockify'
2017-02-09 17:09:51.382 57467 INFO nova.console.websocketproxy [req-f51929d9-8c9b-4df0-abeb-247ce6ef5d65 - - - - -] handler exception: The token '1dfc9af9-8a49-44b3-a955-5196197bc8f7' is invalid or has expired

原因分析

When running a multi node environment with HA between two or more controller nodes(or controller plane service nodes), nova consoleauthservice must be configured with memcached.  
If not, no more than one consoleauth service can berunning in active state, since it need to save the state of the sessions. Whenmemcached is not used, you can check that can connect to the vnc console only afew times when you refresh the page. If that occurs means that the connectionis handled by the consoleauth service that currently is issuing sessions.    
To solve your issue, configure memcached as backend tonova-consoleauth service.  
To solve your issue add this line to nova.conf:  
memcached_servers = 192.168.100.2:11211,192.168.100.3:11211  
This should work to solve your issue.
阅读全文 »

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：

测试中遇到这样的情况，vmware虚拟机中增加的 sdb ,可用空间快要满了，需要增加空间，方法记录如下

1. 在VMware上扩展磁盘

在 vmware 虚拟机编辑页面，扩展磁盘的可用空间 (500GB -> 800GB)
image

2. 重启虚拟机

重启后查看分区是否正确识别，可以看到已经正确识别新的分区大小

# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 859GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  537GB  537GB  primary  xfs

(parted) quit
阅读全文 »
0%