京公网安备11010502063487号在线备份时数据还在写,mongodump 出来的东西能恢复成一个一致的状态吗?--oplog 和 --oplogReplay 到底做了什么?这篇用一个 n == 文档数的不变式逐个验证。
$ mongodump --uri='<副本集>' --db=test --out=/data/m72dump/plain writing `test.c_w` to `/data/m72dump/plain/test/c_w.bson` writing `test.a_counter` to `/data/m72dump/plain/test/a_counter.bson` done dumping `test.a_counter` (1 document) done dumping `test.c_w` (150824 documents) $ mongorestore --uri='<目标>' --drop --nsInclude='test.*' /data/m72dump/plain finished restoring `test.a_counter` (1 document, 0 failures) finished restoring `test.c_w` (150824 documents, 0 failures) 150825 document(s) restored successfully. 0 document(s) failed to restore. 普通 dump 恢复后:a_counter.n = 150,201,c_w 文档数 = 150,824 → 不一致,n - w = -623
写入进行中做普通 mongodump,各集合在不同时间被读取,恢复后 a_counter.n 与 c_w 的文档数差了 623。
$ mongodump --uri='<副本集>' --oplog --out=/data/m72dump/withoplog
writing `admin.system.version` to `/data/m72dump/withoplog/admin/system.version.bson`
done dumping `admin.system.version` (1 document)
writing `test.c_w` to `/data/m72dump/withoplog/test/c_w.bson`
writing `test.a_counter` to `/data/m72dump/withoplog/test/a_counter.bson`
done dumping `test.a_counter` (1 document)
done dumping `test.c_w` (151656 documents)
writing captured oplog to ``
dumped 627 oplog entries
$ ls dump/ dump/test/
admin oplog.bson prelude.json test | test/: a_counter.bson a_counter.metadata.json c_w.bson c_w.metadata.json
oplog.bson 里有 627 条 oplog 记录({'c:applyOps': 627}),时间戳从 1791782833:109 到 1791782837:77,其中对 test.c_w 的 insert 共 627 个、对 test.a_counter 的 update 共 627 个
oplog.bson 第一条记录里的操作(applyOps 展开,ns / op / o / o2): [{"op": "i", "ns": "test.c_w", "o": {"_id": 1035}, "o2": {"_id": 1035}}, {"op": "u", "ns": "test.a_counter", "o": {"$v": 2, "diff": {"u": {"n": 151036}}}, "o2": {"_id": 1}}]--oplog 让 mongodump 在结束时把 dump 期间的 oplog 写进 oplog.bson;事务提交在里面是 applyOps,update 记的是新值。
$ mongorestore --uri='<目标>' --drop --oplogReplay /data/m72dump/withoplog don't know what to do with file `/data/m72dump/withoplog/prelude.json`, skipping... reading metadata for `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.metadata.json` reading metadata for `test.c_w` from `/data/m72dump/withoplog/test/c_w.metadata.json` restoring `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.bson` finished restoring `test.a_counter` (1 document, 0 failures) restoring `test.c_w` from `/data/m72dump/withoplog/test/c_w.bson` finished restoring `test.c_w` (151656 documents, 0 failures) replaying oplog applied 1254 oplog entries --oplogReplay 恢复后:a_counter.n = 151,662,c_w 文档数 = 151,662 → 一致(n == w)
恢复时带 --oplogReplay,mongorestore 导完集合后重放 oplog,1254 个操作后两个集合一致。
$ mongorestore --uri='<目标>' --drop /data/m72dump/withoplog don't know what to do with file `/data/m72dump/withoplog/prelude.json`, skipping... reading metadata for `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.metadata.json` reading metadata for `test.c_w` from `/data/m72dump/withoplog/test/c_w.metadata.json` restoring `test.c_w` from `/data/m72dump/withoplog/test/c_w.bson` restoring `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.bson` finished restoring `test.a_counter` (1 document, 0 failures) finished restoring `test.c_w` (151656 documents, 0 failures) 不带 --oplogReplay 恢复后:a_counter.n = 151,037,c_w 文档数 = 151,656 → 不一致,n - w = -619
不带 --oplogReplay 恢复,oplog.bson 被忽略且没有任何提示,结果和普通 dump 一样不一致。
选取的时间戳 1791782835:78(oplog.bson 第 314/627 条) $ mongorestore --uri='<目标>' --drop --oplogReplay --oplogLimit=1791782835:78 /data/m72dump/withoplog don't know what to do with file `/data/m72dump/withoplog/prelude.json`, skipping... reading metadata for `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.metadata.json` reading metadata for `test.c_w` from `/data/m72dump/withoplog/test/c_w.metadata.json` restoring `test.c_w` from `/data/m72dump/withoplog/test/c_w.bson` restoring `test.a_counter` from `/data/m72dump/withoplog/test/a_counter.bson` finished restoring `test.a_counter` (1 document, 0 failures) finished restoring `test.c_w` (151656 documents, 0 failures) replaying oplog applied 626 oplog entries --oplogLimit 恢复后:a_counter.n = 151,348,c_w 文档数 = 151,656 → 不一致,n - w = -308 对比完整重放:n = 151,662、c_w = 151,662 $ mongorestore --uri='<目标>' --drop --oplogReplay --oplogLimit=1791782837:77 /data/m72dump/withoplog applied 1252 oplog entries --oplogLimit = 最后一条记录的时间戳:a_counter.n = 151,661,c_w 文档数 = 151,661 → 一致(n == w) $ mongorestore --uri='<目标>' --drop --oplogReplay --oplogLimit=1791782837:78 /data/m72dump/withoplog applied 1254 oplog entries --oplogLimit = 最后一条记录的时间戳 + 1:a_counter.n = 151,662,c_w 文档数 = 151,662 → 一致(n == w)
--oplogLimit 取 dump 窗口中间的时间戳会得到不一致的状态(n − w = −308);取窗口末尾才一致,limit 不含时间戳本身。
$ mongodump --uri='<副本集>' --oplog --db=test --out=/data/m72dump/bad1
Failed: bad option: --oplog mode only supported on full dumps
$ mongodump --uri='<目标>' --oplog --out=/data/m72dump/bad2
Failed: error getting oplog start: error getting recent oplog entry: mongo: no documents in result
$ mongorestore --uri='<目标>' --oplogReplay --db=test /data/m72dump/withoplog/test
The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
Failed: cannot use --oplogReplay with includes specified
0 document(s) restored successfully. 0 document(s) failed to restore.--oplog 只支持整库 dump、需要有 oplog 的副本集;--oplogReplay 不能和 --db/includes 一起用。