成功解决:python爬虫http.client.RemoteDisconnected: Remote end closed connection without response

参考:Python 爬虫:http.client.RemoteDisconnected: Remote end closed connection without response 问题解决
使用python爬虫中的urllib 或者 requests 库获取数据时
出现报错:

http.client.RemoteDisconnected: Remote end closed connection without response

原因:
大多数网站都会对用户请求中的 User-Agent 进行检测,如果没有在请求头中设置 User-Agent,那么就会抛出异常
改进方法:在headers中添加:User-Agent

import urllib.request
# 要访问的地址
headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
    try:
        postdata1 = urllib.parse.urlencode(data1).encode("UTF8")
        req_data1 = urllib.request.Request(login_url1, data=postdata1,headers=headers)
        response1 = urllib.request.urlopen(req_data1).read().decode("utf-8", "ignore")
        print(response1)
    except requests.ConnectionError as e:
        print('Error:', e.args)

requests中设置User-Agent如下:

import requests
headers = {
    'User-Agent': "Mozilla/5.0 (Windows; U; Windows NT 5.2) Gecko/2008070208 Firefox/3.0.1"
}
resp = requests.get(url, headers=headers)
print(resp.text)


版权声明:本文为weixin_46569877原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接和本声明。