python从日志文件中提取出现的ip

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


#coding:utf-8
'''
Created on 2015年4月13日

@author: Administrator
'''
fp = open('rootaccess.07log','r')
total_count = 0
ip_array = {}

for eachLine in fp:
    ip =  eachLine.split(" ")[0]
    if ip_array.has_key(ip):
        ip_array[ip] = ip_array[ip] + 1
    else:
        ip_array.setdefault(ip,1)
    total_count = total_count + 1

R=ip_array.items()
num = 0
for i in R:
    if i[1]&gt;0:
        print i
        num+=1
print total_count
fp.close()

日志文件如下所示（160W+行），需要提取首行ip，并统计全部ip数，不包括重复的

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


116.233.77.238 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
61.170.251.177 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
101.86.33.55 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
114.86.53.187 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
180.155.104.206 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
114.88.209.25 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
180.164.251.133 - - [06/Apr/2015:23:59:59 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
116.230.55.212 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
114.95.223.230 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
101.229.143.32 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
114.92.56.189 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
116.230.118.3 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
180.157.88.191 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
124.77.20.187 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
180.155.152.186 - - [07/Apr/2015:00:00:00 +0800] "POST /api/magicboxauth HTTP/1.1" 200 203 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"

如果需要提取全部的ip地址，linux一条命令可以达到要求，但是命令不是排除日志请求中的ip

1

# grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" xxx.log|sort|uniq -c