百度爬虫

1. 目的

使用爬虫脚本 爬去 百度搜索关键字后获得r链接地址以及域名信息
可结合GHDB语法
e.g. inurl:php?id=

2. 知识结构

2.1 使用 threading & queue 模块,多线程处理,自定义线程数
2.2 使用BeautifulSoup & re模块,处理href 匹配
2.3 使用requests 模块,处理web请求&获得请求后的真实地址(r.url)
2.4 百度最大搜索页面76页,pn max 760
2.5 将结果存入文本,域名已去重

3. 爬虫脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#coding=utf-8
import requests
import re
import Queue
import threading
from bs4 import BeautifulSoup as bs
import os,sys,time
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'}
class BaiduSpider(threading.Thread):
def __init__(self,queue):
threading.Thread.__init__(self)
self._queue = queue
def run(self):
while not self._queue.empty():
url = self._queue.get_nowait()
try:
#print url
self.spider(url)
except Exception,e:
print e
pass
def spider(self,url):
#if not add self , error:takes exactly 1 argument (2 given)
r = requests.get(url=url,headers=headers)
soup = bs(r.content,'lxml')
urls = soup.find_all(name='a',attrs={'data-click':re.compile(('.')),'class':None})
for url in urls:
#print url['href']
new_r = requests.get(url=url['href'],headers=headers,timeout=3)
if new_r.status_code == 200 :
url_para = new_r.url
url_index_tmp = url_para.split('/')
url_index = url_index_tmp[0]+'//'+url_index_tmp[2]
print url_para+'\n'+url_index
with open('url_para.txt','a+') as f1:
f1.write(url_para+'\n')
with open('url_index.txt','a+') as f2:
with open('url_index.txt', 'r') as f3:
if url_index not in f3.read():
f2.write(url_index+'\n')
else:
print 'no access',url['href']
def main(keyword):
queue = Queue.Queue()
de_keyword = keyword.decode(sys.stdin.encoding).encode('utf-8')
print keyword
# baidu max pages 76 , so pn=750 max
for i in range(0,760,10):
#queue.put('https://www.baidu.com/s?ie=utf-8&wd=%s&pn=%d'%(keyword,i))
queue.put('https://www.baidu.com/s?ie=utf-8&wd=%s&pn=%d'%(de_keyword,i))
threads = []
thread_count = 4
for i in range(thread_count):
threads.append(BaiduSpider(queue))
for t in threads:
t.start()
for t in threads:
t.join()
if __name__ == '__main__':
if len(sys.argv) != 2:
print 'Enter:%s keyword'%sys.argv[0]
sys.exit(-1)
else:
main(sys.argv[1])

效果图

4. 待优化点

4.1 多个搜索引擎的处理
4.2 多参数的处理
4.2 payload 结合

5. 参考信息

5.1. ADO ichunqiu Python安全工具开发应用
5.2. https://github.com/sharpdeep/CrawlerBaidu/blob/master/CrawlerBaidu.py