京公网安备11010502063487号Scrapy 的信号有哪些、什么时候发、带什么参数?处理函数抛异常、写成 async 会怎样?能不能用信号提前中止一次下载?这篇给全部 22 个信号接上记录器,在一次包含各种情形的抓取里量出来。
scrapy.signals 里有 22 个信号:bytes_received engine_started engine_stopped feed_exporter_closed feed_slot_closed headers_received item_dropped item_error item_scraped memusage_warning_reached request_dropped request_left_downloader request_reached_downloader request_scheduled response_downloaded response_received robots_parsed scheduler_empty spider_closed spider_error spider_idle spider_opened 信号 次数 参数(除 sender、signal) bytes_received 6 ['data', 'request', 'spider'] engine_started 1 [] engine_stopped 1 [] feed_exporter_closed 1 [] feed_slot_closed 1 ['slot'] headers_received 7 ['body_length', 'headers', 'request', 'spider'] item_dropped 1 ['exception', 'item', 'response', 'spider'] item_error 1 ['failure', 'item', 'response', 'spider'] item_scraped 2 ['item', 'response', 'spider'] memusage_warning_reached 0 (没触发) request_dropped 1 ['request', 'spider'] request_left_downloader 7 ['request', 'spider'] request_reached_downloader 7 ['request', 'spider'] request_scheduled 7 ['request', 'spider'] response_downloaded 7 ['request', 'response', 'spider'] response_received 6 ['request', 'response', 'spider'] robots_parsed 1 ['request', 'robotparser'] scheduler_empty 3 [] spider_closed 1 ['reason', 'spider'] spider_error 1 ['failure', 'response', 'spider'] spider_idle 1 ['spider'] spider_opened 1 ['spider']
请求生命周期的四个信号(scheduled、reached_downloader、left_downloader、response_downloaded)每个下载的请求各触发一次;response_received 比它们少,因为被重定向这类中间件消费的响应不会走到引擎。
首次出现的顺序:spider_opened → engine_started → request_scheduled → request_reached_downloader → headers_received → bytes_received → response_downloaded → request_left_downloader → response_received → robots_parsed → scheduler_empty → request_dropped → spider_error → item_scraped → item_dropped → item_error → spider_idle → spider_closed → feed_slot_closed → feed_exporter_closed → engine_stopped 最后 6 个事件:scheduler_empty → spider_idle → spider_closed → feed_slot_closed → feed_exporter_closed → engine_stopped
每个请求的信号链是 scheduled → reached → headers_received → bytes_received → response_downloaded → left → response_received;结束时 spider_idle → spider_closed → feed_* → engine_stopped。
StopDownload(fail=False):[{'sig': 'bytes_received', 'n': 65536}, {'who': 'callback', 'body': 65536, 'status': 200}];stats downloader/response_bytes=65670
StopDownload(fail=True):[{'sig': 'bytes_received', 'n': 65536}, {'who': 'errback', 'exc': 'StopDownload', 'resp': 65536}];stats downloader/response_bytes=None
不中止:[{'who': 'callback', 'body': 300000, 'status': 200}];bytes_received 触发了 5 次;response_bytes=300134headers_received 和 bytes_received 的处理函数可以抛 StopDownload,fail 参数决定把截断的响应交给回调还是 errback。
信号 async 处理函数 spider_opened 被 await 到结束 engine_started 被 await 到结束 request_scheduled 没有运行(协程没被调度) request_reached_downloader 没有运行(协程没被调度) headers_received 没有运行(协程没被调度) bytes_received 没有运行(协程没被调度) response_downloaded 没有运行(协程没被调度) request_left_downloader 没有运行(协程没被调度) response_received 没有运行(协程没被调度) item_scraped 被 await 到结束 scheduler_empty 没有运行(协程没被调度) spider_idle 没有运行(协程没被调度) spider_closed 被 await 到结束 feed_slot_closed 被 await 到结束 feed_exporter_closed 被 await 到结束 engine_stopped 被 await 到结束 日志:['[scrapy.utils.signal] ERROR: Error caught on signal handler: <function S.from_crawler.<locals>.bad a'];finish_reason=finished(处理函数抛异常没有影响)
处理函数异常只记日志。async 处理函数只在 Scrapy 用 send_catch_log_async 触发的信号上被 await,其它信号上等于不运行。
不保存引用的局部函数:处理函数被调用 0 次 把函数挂到 spider 属性上:处理函数被调用 1 次
connect 默认只存弱引用,局部函数要自己保存引用。