爬虫——爬取晋江免费文章

主要目的是为了记录学习爬虫过程。

<code># -*- coding: UTF-8 -*-
import requests
import re
import json
from multiprocessing import Pool
from requests.exceptions import RequestException
import itertools
headers = {'Accept':'*/*',
'Accept-Encoding':   'gzip, deflate',
'Accept-Language':    'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
'Connection':   'keep-alive',
'Cookie':    'cna=um0tE7tQllkCAXE21MrqNDM2',
'DNT':    1,
'Host':    'afp.csbew.com',
'User-Agent':    'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0'}

def get_one_page(url):
    
    response = requests.get(url,params = headers)
    response.encoding = "gb2312"
    html = response.text
    # html = html.encode(encoding="gbk", errors="ignore").decode("gbk", errors="ignore")
    html = html.encode(encoding="utf-8", errors="ignore").decode("utf-8", errors="ignore")
    return html

def parse_one_page(html):
    pattern = re.compile('.*?class="noveltext">.*?<h2>(.*?)</h2>.*?both;"></div>(.*?)<div id="favoriteshow_3"',re.S)
    items = re.findall(pattern,html)
    for item in items:
        yield{
            'chapter':item[0],
            'contents':item[1]
            }

def write_to_file(text):
    with open('novel.txt','a',encoding='utf-8') as f:
        f.write(json.dumps(text,ensure_ascii=False)+'\n')
        
    
    
def main(novelid , num ):
    url='http://www.jjwxc.net/onebook.php?novelid='+str(novelid)+'&chapterid='+str(num)
    print(url)
    html=get_one_page(url)
    for item in parse_one_page(html):
        write_to_file(item)
    print('ok')
    
def main_vice(novelid):
    for x in range(16):
        print(x,novelid)
        main(novelid,x+1)      


if __name__ == '__main__':
    pool = Pool(3)
    with open('rank.txt','r',encoding='utf-8') as file:
        novelid = []
        for i in file.readlines():
            file = json.loads(i)
            for key,value in file.items():
                novelid.append(value)

    pool.map(main_vice,novelid)</code>

<p>思路就是简单的观察晋江文章网站的网页地址，可得规律，然后用正则表达式解析html，获得文章内容，写入文件中。

代码有一个问题，就是得到的文章中有很多换行，可以用.replace替换，运行较慢。同时，由于文章id是不太具有规律性的，因为我下载的是月榜内容，所以，得写一个文件来收集排行榜上的文章id。</p>