京公网安备11010502063487号爬虫中间件夹在引擎和 spider 之间,管的是“进入回调的响应”和“回调产出的请求与 item”。这篇用三个自定义中间件加记录钩子,把输入、输出、异常三条链的方向与短路规则一条条实测出来。
StartSpiderMiddleware(25) → HttpErrorMiddleware(50) → RefererMiddleware(700) → UrlLengthMiddleware(800) → DepthMiddleware(900) → MetaCopyDetectionMiddleware(1000)
1. A process_start
2. B process_start
3. C process_start
4. A process_spider_input
5. B process_spider_input
6. C process_spider_input
7. spider callback url=list
8. C process_spider_output what=item {'n': 1}
9. B process_spider_output what=item {'n': 1}
10. A process_spider_output what=item {'n': 1}
11. C process_spider_output what=Request second
12. B process_spider_output what=Request second
13. A process_spider_output what=Request second
14. A process_spider_input
15. B process_spider_input
16. C process_spider_input
17. spider callback url=second响应进来走 process_spider_input(数字小 → 大),回调产出走 process_spider_output(数字大 → 小),process_start 处理 start() 的输出。
1. A process_start 2. B process_start 3. C process_start 4. A process_spider_input 5. B process_spider_input 6. spider errback exc=ValueError: input rejected by B spider_exceptions/count=0
在 process_spider_input 里抛异常会跳过回调和更靠后的 input 方法,Failure 直接交给请求的 errback。HttpErrorMiddleware 就是这么拒绝非 2xx 响应的。
1. A process_start
2. A process_spider_input
3. A process_spider_output what=Request i0
4. A process_spider_output what=Request i1
5. A 丢弃 what=Request i1
6. A process_spider_output what=Request i2
7. A process_spider_output what=Request i3
8. A 丢弃 what=Request i3
9. A process_spider_output what=item {'price': 5}
10. A process_spider_output what=item {'price': -1}
11. A 丢弃 what=item {'price': -1}
12. A process_spider_input
13. spider callback url=i2
14. A process_spider_input
15. spider callback url=i0
服务器收到 ['/list', '/i2', '/i0'];item_scraped_count=1process_spider_output 对每个请求和 item 单独决定放行与否,丢弃的对象不会进入调度器或 pipeline。
没有人恢复:
1. A process_spider_input
2. B process_spider_input
3. C process_spider_input
4. spider callback url=list
5. C process_spider_output what=item {'n': 1}
6. B process_spider_output what=item {'n': 1}
7. A process_spider_output what=item {'n': 1}
8. C process_spider_exception exc=RuntimeError
9. B process_spider_exception exc=RuntimeError
10. A process_spider_exception exc=RuntimeError
spider_exceptions/count=1;item_scraped_count=1
B(300)恢复:process_spider_exception 里返回 [{'recovered_by': 'B'}]:
1. A process_spider_input
2. B process_spider_input
3. C process_spider_input
4. spider callback url=list
5. C process_spider_output what=item {'n': 1}
6. B process_spider_output what=item {'n': 1}
7. A process_spider_output what=item {'n': 1}
8. C process_spider_exception exc=RuntimeError
9. B process_spider_exception exc=RuntimeError
10. A process_spider_output what=item {'recovered_by': 'B'}
spider_exceptions/count=0;item_scraped_count=2回调抛异常后,各中间件的 process_spider_exception 依次被调用:返回 None 交给下一个,返回可迭代对象则终止异常链,内容只经过更靠近引擎的 output。
只有同步版本的 Old: TypeError: Middleware spider.Old doesn't support asynchronous spider output. Its process_spider_output() method should be an async generator function or it should additionally define a process_spider_output_asy
同步 + process_spider_output_async 的 New: 正常运行,item_scraped_count=1,[{'what': 'async 版本被调用', 't': 161.3}]2.19 不接受只有同步 process_spider_output 的中间件:要写成 async 生成器,或再定义 process_spider_output_async。
回调被调用的:['ok'];服务器收到 ['/404', '/ok'] stats:downloader/response_status_count/200=1, downloader/response_status_count/404=1, httperror/response_ignored_count=1, httperror/response_ignored_status_count/404=1, response_received_count=2, urllength/request_ignored_count=1 日志:[scrapy.spidermiddlewares.urllength] INFO: Ignoring link (url length > 2083): http://127.0.0.1:9960/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 日志:[scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://127.0.0.1:9960/404>: HTTP status code is not handled or not allowed
HttpError 过滤非 2xx 响应,UrlLength 丢弃过长的 URL,都会在 stats 里计数。