京公网安备11010502063487号爬虫日志里最常见的错误是一长串“Connection aborted”“Max retries exceeded”。这篇用我写的套接字服务器逐种制造:对端礼貌地关、直接重置、回了乱码、响应头太多、半截响应,看 requests 到底抛什么,最里层的原因是什么,又会不会被 except ConnectionError 接住。
没有任何服务在监听(端口被拒绝)
requests 抛出:ConnectionError;内部链:ConnectionError → MaxRetryError → NewConnectionError → ConnectionRefusedError;最内层:[Errno 61] Connection refused;ConnectionError/Timeout = 是/否
收到请求后,读完并正常关闭连接(FIN)
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → RemoteDisconnected;最内层:Remote end closed connection without response;ConnectionError/Timeout = 是/否
请求字节到达后,不读直接关闭(缓冲区有未读数据,系统发 RST)
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → ConnectionResetError;最内层:[Errno 54] Connection reset by peer;ConnectionError/Timeout = 是/否
读完请求后,设 SO_LINGER=0 强制关闭(RST)
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → ConnectionResetError;最内层:[Errno 54] Connection reset by peer;ConnectionError/Timeout = 是/否
只发了半截响应头就关闭
requests 抛出:(无异常);内部链:状态 200;最内层:;ConnectionError/Timeout =
回了不是 HTTP 的内容
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → BadStatusLine;最内层:hello world\r\n;ConnectionError/Timeout = 是/否
状态行里状态码不是数字
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → BadStatusLine → ValueError;最内层:invalid literal for int() with base 10: 'abc';ConnectionError/Timeout = 是/否
一条响应头长度 70000 字节
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → LineTooLong;最内层:got more than 65536 bytes when reading header line;ConnectionError/Timeout = 是/否
响应头 150 个
requests 抛出:ConnectionError;内部链:ConnectionError → ProtocolError → HTTPException;最内层:got more than 100 headers;ConnectionError/Timeout = 是/否
Content-Length: 1000 只发 100 字节就关闭
requests 抛出:ChunkedEncodingError;内部链:ChunkedEncodingError → ProtocolError → IncompleteRead;最内层:IncompleteRead(100 bytes read, 900 more expected);ConnectionError/Timeout = 否/否
Content-Length: 1000 发 100 字节后发 RST
requests 抛出:ChunkedEncodingError;内部链:ChunkedEncodingError → ProtocolError → ConnectionResetError;最内层:[Errno 54] Connection reset by peer;ConnectionError/Timeout = 否/否
(细看"只发了半截响应头就关闭":requests 没有报错,r.status_code=200,收到的响应头={},r.content=b'')连接与响应头阶段的异常对外都是 requests.ConnectionError,要看内部链最里层才知道原因:没人监听是 ConnectionRefusedError;服务器读完请求后正常关闭(FIN)是 RemoteDisconnected;关闭时缓冲区里有未读数据或设了 SO_LINGER=0(RST)是 ConnectionResetError。第 4 篇里的竞态就是 RemoteDisconnected。
响应不合规时:不是 HTTP 是 BadStatusLine;一条响应头超过 65536 字节是 LineTooLong;响应头超过 100 个是 HTTPException。响应体读到一半断开(长度不足或被重置)是 ChunkedEncodingError。
最值得注意的是半截响应头:服务器发了 HTTP/1.1 200 OK\r\nContent-Le 就关闭,requests 返回 200、空头、空内容,没有任何异常。爬虫如果只看状态码,就会把这种半截响应当成功。
https:// 去连明文 HTTP 服务器:抛 SSLError;内部链 SSLError → MaxRetryError → SSLError → SSLError;最内层 [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1082);ConnectionError/Timeout = 是/否 http:// 去连一个 TLS 服务器:抛 ConnectionError;内部链 ConnectionError → ProtocolError → ConnectionResetError;最内层 [Errno 54] Connection reset by peer;ConnectionError/Timeout = 是/否 https:// 连自签名证书的服务器且 verify=True:抛 SSLError;内部链 SSLError → MaxRetryError → SSLError → SSLCertVerificationError;最内层 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certific;ConnectionError/Timeout = 是/否
https 地址连到明文端口得到 SSLError(WRONG_VERSION_NUMBER);http 地址连到 TLS 端口,服务器看到的是乱码,这里得到 ConnectionResetError;自签名证书加 verify=True 是 SSLError(CERTIFICATE_VERIFY_FAILED)。这三种都是 ConnectionError 的子类。
ConnectionError:ConnectionError → RequestException → OSError → Exception ConnectTimeout:ConnectTimeout → ConnectionError → Timeout → RequestException → OSError → Exception ReadTimeout:ReadTimeout → Timeout → RequestException → OSError → Exception SSLError:SSLError → ConnectionError → RequestException → OSError → Exception ProxyError:ProxyError → ConnectionError → RequestException → OSError → Exception ChunkedEncodingError:ChunkedEncodingError → RequestException → OSError → Exception ContentDecodingError:ContentDecodingError → RequestException → OSError → HTTPError → Exception TooManyRedirects:TooManyRedirects → RequestException → OSError → Exception InvalidURL:InvalidURL → RequestException → OSError → ValueError → Exception
ConnectTimeout、SSLError、ProxyError 继承自 ConnectionError;ReadTimeout 只继承 Timeout;ChunkedEncodingError 既不是 ConnectionError 也不是 Timeout。所以只写 except ConnectionError 会漏掉读超时和响应体阶段的断开;想统一处理要捕获 requests.RequestException。