京公网安备11010502063487号分片集群上的聚合是怎么跑的?哪些阶段在分片上执行,哪些在合并阶段?这篇用 explain 的 splitPipeline 逐个情形看。
块分布: [(MinKey(), 10000, 'sh0'), (10000, 20000, 'sh1'), (20000, MaxKey(), 'sh2')]
$match uid = 15000 {"shards": ["sh1"], "splitPipeline": null, "keys": ["$clusterTime", "command", "ok", "operationTime", "queryShapeHash", "serverInfo"]}
$match uid 范围 [0, 5000) + $group {"shards": ["sh0"], "splitPipeline": null, "keys": ["$clusterTime", "command", "ok", "operationTime", "queryShapeHash", "serverInfo"]}$match 里有分片键条件时,mongos 只把 pipeline 发给相关分片(uid = 15000 只命中 sh1,uid < 5000 只命中 sh0),explain 里没有 splitPipeline,整条 pipeline 在那个分片上执行。
$group by city {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$group"], "mergerPart": ["$mergeCursors", "$group"]}
$group by city, 然后 $sort {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$group"], "mergerPart": ["$mergeCursors", "$group", "$sort"]}没有分片键条件的 $group 访问全部 3 个分片:shardsPart = [$group],mergerPart = [$mergeCursors, $group];随后的 $sort 也在 mergerPart。
sh0:扫描 10,000 个文档,返回给合并阶段 50 个文档 sh1:扫描 10,000 个文档,返回给合并阶段 50 个文档 sh2:扫描 10,000 个文档,返回给合并阶段 50 个文档
每个分片扫描 10,000 个文档,只返回 50 个分好组的文档,合并阶段一共收到 150 个,而不是 30,000 个。
$sort amount, uid; $limit 5 {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$sort"], "mergerPart": ["$mergeCursors", "$limit"]}
shardsPart[0] = {"$sort": {"sortKey": {"amount": -1, "uid": 1}, "limit": 5}}
mergerPart = ['$mergeCursors', '$limit'] $mergeCursors 的 sort = {"amount": -1, "uid": 1} $mergeCursors 的字段 = ['allowPartialResults', 'compareWholeSortKey', 'lsid', 'nss', 'requestQueryStatsFromRemotes', 'sort', 'tailableMode']limit 被并进分片上的 $sort(limit: 5);mergerPart 里 $mergeCursors 带着同样的 sort 做有序归并,再 $limit。
$group + $lookup(cities) {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$group"], "mergerPart": ["$mergeCursors", "$group", "$lookup"]}
$match + $out {"shards": ["sh0"], "splitPipeline": null, "keys": ["$clusterTime", "command", "ok", "operationTime", "queryShapeHash", "serverInfo"]}
$group + $merge {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$group"], "mergerPart": ["$mergeCursors", "$group", "$merge"]}$group 之后的 $lookup 和 $merge 排在 mergerPart 里;$match uid < 5000 + $out 只命中 sh0,在该分片上整条执行。
只有 $group:3.9 ms;$sort 在 $group 前(排序整个集合再分组):18 ms
后一种的拆分: {"shards": ["sh0", "sh1", "sh2"], "shardsPart": ["$sort", "$project"], "mergerPart": ["$mergeCursors", "$group"]}只有 $group 的聚合 3.9 ms;$group 前面加 $sort 后拆分变成 shardsPart = [$sort, $project],$group 整个落到合并阶段,耗时 18 ms。