京公网安备11010502063487号默认请求头从哪来?设置、请求、spider 里都能写 User-Agent,谁最终生效?发出的头是什么顺序和大小写?这篇用原始套接字服务器看真正出现在网络上的字节。
GET 请求:
GET /a HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
Host: 127.0.0.1:端口DefaultHeadersMiddleware(DEFAULT_REQUEST_HEADERS)、UserAgentMiddleware(USER_AGENT)和 HttpCompressionMiddleware(Accept-Encoding)各自用 setdefault 补头;请求自己的头永远优先。
USER_AGENT = Scrapy/2.19.0 (+https://scrapy.org)
DEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en'}Accept 和 Accept-Language 在 DEFAULT_REQUEST_HEADERS 里,USER_AGENT 带着版本号;Accept-Encoding 不在设置里。
只改 USER_AGENT 设置:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: MyBot/1.0
Accept-Encoding: gzip, deflate, br, zstd
Request 的 headers 里写 User-Agent:
User-Agent: PerRequest/9
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
Accept-Encoding: gzip, deflate, br, zstd
DEFAULT_REQUEST_HEADERS 改 Accept-Language,新增 X-Token:
Accept: application/json
Accept-Language: zh-CN
X-Token: t
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
Request 的 headers 覆盖 Accept-Language:
Accept-Language: fr
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
USER_AGENT 设成空串:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
Accept-Encoding: gzip, deflate, br, zstd
(没有 User-Agent 头)
COMPRESSION_ENABLED=0:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)替换 DEFAULT_REQUEST_HEADERS 是整体替换;把 USER_AGENT 设空串则不发这个头;关闭压缩中间件则不发 Accept-Encoding。
GET /a HTTP/1.1
User-Agent: lower
Accept: UPPER
X-Custom-Header: v
X-Multi: a
X-Multi: b
Accept-Language: en
Accept-Encoding: gzip, deflate, br, zstd
Host: 127.0.0.1:端口头名在发出前规范成首字母大写,值是列表时拆成多行。
POST /p HTTP/1.1
Content-Length: 7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
Host: 127.0.0.1:端口
a=1&b=2
FormRequest(formdata=…):
POST /p HTTP/1.1
Content-Length: 7
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: Scrapy/2.19.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
Host: 127.0.0.1:端口
a=1&b=2手写 POST 的 Request 只有 Content-Length;FormRequest 会加上表单的 Content-Type。