request.meta 里的特殊键

  共 45 个键,其中公开 40 个、内部(下划线)5 个
  公开键:allow_offsite autothrottle_dont_adjust_delay bindaddress cache_timestamp cached_response cookiejar depth depth_reset dont_cache dont_merge_cookies dont_obey_robotstxt dont_redirect dont_retry download_latency download_maxsize download_slot download_timeout download_warnsize ftp_local_filename ftp_passive ftp_password ftp_user handle_httpstatus_all handle_httpstatus_list http_auth_domain http_pass http_user is_secure is_start_request max_retry_times priority_adjust proxy redirect_reasons redirect_times redirect_ttl redirect_urls referrer_policy retry_times rule verbatim_url
  内部键:_auth_proxy _callback _depth _dont_cache _scheme_proxy
  按用途分:你设置的输入键 24 个(allow_offsite autothrottle_dont_adjust_delay bindaddress cookiejar depth_reset dont_cache dont_merge_cookies dont_obey_robotstxt dont_redirect dont_retry download_maxsize download_slot download_timeout download_warnsize handle_httpstatus_all handle_httpstatus_list http_auth_domain http_pass http_user max_retry_times priority_adjust proxy referrer_policy verbatim_url);Scrapy 写入的输出键 10 个(cache_timestamp depth download_latency is_start_request redirect_reasons redirect_times redirect_ttl redirect_urls retry_times rule)
  源码里出现但上面两组都没列的公开键:cached_response ftp_local_filename ftp_passive ftp_password ftp_user is_secure

  /big [('download_maxsize', 1000)]                           → errback DownloadCancelledError
  /big []                                                     → 回调 status=200 5000 字节
  /e404 [('handle_httpstatus_all', True)]                     → 回调 status=404 1 字节
  /e404 []                                                    → errback HttpError
  /e500 [('handle_httpstatus_list', [500])]                   → 回调 status=500 1 字节
  /e503 [('handle_httpstatus_list', [500])]                   → errback HttpError
  /slow [('download_timeout', 0.3)]                           → errback DownloadTimeoutError
  /slow []                                                    → 回调 status=200 18 字节

  不设                                  → [{'who': 'callback'}];服务器看到的客户端地址 127.0.0.1
  ('127.0.0.1', 0)                    → [{'who': 'callback'}];服务器看到的客户端地址 127.0.0.1
  ('10.255.255.1', 0)(本机没有这个地址)       → [{'who': 'errback', 'exc': 'ConnectBindError', 'msg': "Couldn't bind: 49: Can't assign requested address."}];服务器看到的客户端地址 (没收到)

  None                              → 目标请求的 Referer = http://主机/link?q=1
  no-referrer                       → 目标请求的 Referer = (无)
  origin                            → 目标请求的 Referer = http://主机/
  same-origin                       → 目标请求的 Referer = http://主机/link?q=1
  strict-origin-when-cross-origin   → 目标请求的 Referer = http://主机/link?q=1
  unsafe-url                        → 目标请求的 Referer = http://主机/link?q=1

  meta http_user/http_pass:/a → alice:s3cret;加 http_auth_domain=localhost(请求在 127.0.0.1)/b → (无);请求自带 Authorization 头 /c → ['Bearer own']
  只设 HTTPAUTH_USER / HTTPAUTH_PASS(没设 HTTPAUTH_DOMAIN):退出码≠0?是;服务器收到 0 个请求;错误:ValueError: HTTPAUTH_DOMAIN must be set when HTTPAUTH_USER or HTTPAUTH_PASS is configured. Set it to a domain (e.g. 'example.com') to restrict credent
  HTTPAUTH_DOMAIN=127.0.0.1(不带端口,请求 URL 带端口):/a → (无)
  HTTPAUTH_DOMAIN=localhost(请求在 127.0.0.1):/a → (无)
  HTTPAUTH_DOMAIN=127.0.0.1:<端口>(域名带上端口):/a → bob:pw  ← 匹配用的是 URL 的 netloc(含端口),所以 URL 里有端口时域名也得带端口
  meta http_auth_domain=127.0.0.1:<端口>:/b → alice:pw

  response.meta 共 10 个键:['download_latency', 'download_slot', 'download_timeout', 'is_start_request', 'mine', 'redirect_reasons', 'redirect_times', 'redirect_ttl', 'redirect_urls', 'retry_times']
  其中的取值:{'mine': 'x', 'is_start_request': True, 'download_timeout': '…', 'download_slot': '127.0.0.1', 'download_latency': '…', 'redirect_times': 1, 'redirect_ttl': 19, 'redirect_urls': '…', 'redirect_reasons': [302], 'retry_times': 1}