京公网安备11010502063487号同一个 GET 请求,不同客户端发出的字节并不一样。服务器看得到你发了哪些头、什么顺序、什么大小写。这篇用一个套接字服务器抓下 requests、urllib3、http.client、aiohttp、curl、wget、curl_cffi、Node、真实的 Chrome 发出的原始请求头,再看看在 requests 里自己改头会发生什么。
requests:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
User-Agent: python-requests/2.34.2
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive
urllib3:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
Accept-Encoding: identity
User-Agent: python-urllib3/2.7.0
http.client:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
Accept-Encoding: identity
aiohttp:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
Accept: */*
Accept-Encoding: gzip, deflate, zstd
User-Agent: Python/3.14 aiohttp/3.14.3
curl 命令行:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
User-Agent: curl/8.7.1
Accept: */*
wget:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
User-Agent: Wget/1.25.0
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive
curl_cffi:GET / HTTP/1.1
Host: 127.0.0.1:<端口>
Accept-Encoding: gzip, deflate, br
Node 的 fetch:GET / HTTP/1.1
host: 127.0.0.1:<端口>
connection: keep-alive
accept: */*
accept-language: *
sec-fetch-mode: cors
user-agent: node
accept-encoding: gzip, deflate
Chrome(headless):GET / HTTP/1.1
Host: 127.0.0.1:<端口>
Connection: keep-alive
sec-ch-ua: "Chromium";v="154", "Google Chrome";v="154", ...
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_...
Accept: text/html,application/xhtml+xml,application/x...
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9HTTP/1.1 里头名不分大小写,但服务器看得到你发的原始字节。requests 默认发 5 个头;urllib3 和 http.client 更少(http.client 只有 Host 和 Accept-Encoding: identity);curl_cffi 默认也只有 Host 和 Accept-Encoding;Node 的 fetch 所有头名都是小写;真实的 Chrome 154 发 14 个,包括 sec-ch-ua*、Sec-Fetch-*、Upgrade-Insecure-Requests、Accept-Language。
requests:Host, User-Agent, Accept-Encoding, Accept, Connection urllib3:Host, Accept-Encoding, User-Agent http.client:Host, Accept-Encoding aiohttp:Host, Accept, Accept-Encoding, User-Agent curl 命令行:Host, User-Agent, Accept wget:Host, User-Agent, Accept, Accept-Encoding, Connection curl_cffi:Host, Accept-Encoding Node 的 fetch:host, connection, accept, accept-language, sec-fetch-mode, user-agent, accept-encoding Chrome(headless):Host, Connection, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Upgrade-Insecure-Requests, User-Agent, Accept, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-User, Sec-Fetch-Dest, Accept-Encoding, Accept-Language 9 个客户端里,头名(忽略大小写)的顺序彼此不同的有 8 种;区分大小写后有 8 种 顺序完全相同(忽略大小写)的客户端:['http.client', 'curl_cffi'] Host 头是第一个头的客户端:['requests', 'urllib3', 'http.client', 'aiohttp', 'curl 命令行', 'wget', 'curl_cffi', 'Node 的 fetch', 'Chrome(headless)'] 头名全是小写的客户端:['Node 的 fetch']
9 个客户端里,头名顺序彼此不同的有 8 种;只有 http.client 和 curl_cffi 恰好相同。全部以 Host 开头。Chrome 的头名大小写混合(sec-ch-ua* 小写,其余首字母大写)。服务器如果愿意,可以用这些特征区分客户端——这是为什么“把 User-Agent 改成浏览器的”并不能让请求看起来像浏览器。
默认:['Host', 'User-Agent: python-requests/2.34.2', 'Accept-Encoding: gzip, deflate, zstd', 'Accept', 'Connection']
headers={"x-custom": "1", "User-Agent": "my"}:['Host', 'User-Agent: my', 'Accept-Encoding: gzip, deflate, zstd', 'Accept', 'Connection', 'x-custom']
headers={"user-agent": "my"}(小写名字覆盖默认的 User-Agent):['Host', 'user-agent: my', 'Accept-Encoding: gzip, deflate, zstd', 'Accept', 'Connection']
headers={"Accept": "text/html", "User-Agent": "my", "Accept-Encoding": "gzip"}(想改顺序):['Host', 'User-Agent: my', 'Accept-Encoding: gzip', 'Accept', 'Connection']
headers={"User-Agent": None}(去掉默认头):['Host', 'User-Agent: python-urllib3/2.7.0', 'Accept-Encoding: gzip, deflate, zstd', 'Accept', 'Connection']
headers={"Connection": None, "Accept-Encoding": None, "Accept": None}:['Host', 'Accept-Encoding: identity', 'User-Agent: python-requests/2.34.2']
Session.headers.clear() 后只加 Accept:['Host', 'Accept-Encoding: identity', 'User-Agent: python-urllib3/2.7.0', 'Accept']
同时给出 "X-A" 和 "x-a" 两个同名头(大小写不同):['Host', 'User-Agent: python-requests/2.34.2', 'Accept-Encoding: gzip, deflate, zstd', 'Accept', 'Connection', 'x-a: 2']自己加的头排在默认头之后;默认头的位置固定,即使你传入的字典顺序不同也不会重排;用小写名字覆盖默认头,位置不变但名字的大小写按你给的来。把某个默认头设成 None 不一定能去掉:User-Agent 被去掉后,下层的 urllib3 又补上了自己的 python-urllib3/2.7.0;Accept-Encoding 去掉后 http.client 补上 identity。大小写不同的同名头在请求里只保留最后一个。