爬虫中文乱码解决方法总结

第一种方法

import requests
url = 'http://www.***.com'
headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
html_temp = requests.get(url, headers=headers)
html_temp .encoding = html_temp .apparent_encoding
html = html_temp .text
print(html)

此时的获取到的HTML源码就是正常的中文文本

第二种方法

import requests
url = 'http://www.***.com'
headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
html= requests.get(url, headers=headers).content.decode('utf-8')
print(html)

备注：若是用第二种方法报错说utf-8不能解析网站，则此时就只需要更改编码（具体改成哪种格式根据爬取网站的编码来，一般网站大部分都是UTF-8或者GBK）,所以可以改成gbk试下

html= requests.get(url, headers=headers).content.decode('GBK')

第三种方法

import requests
url = 'http://www.***.com'
headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
content = requests.get(url, headers=headers)
content.encoding = 'utf-8'
html= content.text
print(html)

Unicode编码转中文

def spider(page):
    url = ‘**********’
    content = requests.get(url).content.decode('unicode-escape')
    print(content)