MongoDB point-in-time backup and recovery.

In this blog, we describe a procedure that allows you to manually perform a point-in-time restore of MongoDB data. Point-in-time restore refers to the ability to restore the database to a precise moment in time. The instructions presented here require that you have regular backups of your data, and that your data is stored in a replica set.

Requirements :mongodb22

  1. You need a backup of the database – this can be either from a mongodump or from a file system copy.
  2. You need access to an oplog that contains data that goes back to when the backup was made. For example if you made a backup 24 hours ago, then you need an oplog that goes back at least 24 hours. This oplog will be extracted from one of the replica set members in the live database.

Mongodump is the base logical backup tool included with MongoDB. It takes a full BSON copy of database/collections, and optionally includes a log of changes during the backup used to make it consistent to a point in time. Mongorestore is the tool used to restore logical backups created by Mongodump. I’ll use these tools in the steps in this article to restore backed-up data. This article assumes a mongodump-based backup that was taken consistently with oplog changes (by using the command flag “–oplog”), and the backup is being restored to a MongoDB instance.

Example scenario :

We have a backup of mongodb backup of databases running in regular intervals.

mongodump

Assuming you have oplog on your live/production database that goes back at least 24 hours (ie. an oplog window of at least 24 hours). The oplog is available on each node in the replica set, and can be dumped out from any one of the replica set members. The oplog contains a record of all database events that have recently occurred.

For testing we are added data to the collection and verified the current document count

rs0:PRIMARY> use testdata
switched to db testdata
rs0:PRIMARY> db.birthday.find().count()
2000
rs0:PRIMARY> db.birthday.find().limit(1)
{ "_id" : ObjectId("5bf18634df205e1d523542fa"), "name" : "name1", "birthday" : ISODate("1974-09-18T00:00:00Z") }
rs0:PRIMARY> db.birthday.insertOne({name: "name99", birthday: ISODate("1990-02-11T02:00:00Z") }
... )
{
        "acknowledged" : true,
        "insertedId" : ObjectId("5bf19caeeded9b5c7021d2f3")
}
rs0:PRIMARY> db.birthday.find().limit(1)
{ "_id" : ObjectId("5bf18634df205e1d523542fa"), "name" : "name1", "birthday" : ISODate("1974-09-18T00:00:00Z") }
rs0:PRIMARY> db.birthday.find().count()
2001

datainsert1

Now we are dropping the collection birthday

dropcoll

Restoration instructions:

  1. Verify the last time stamp in oplog
rs0:PRIMARY> use local
switched to db local
rs0:PRIMARY> db.oplog.rs.find({}, {ts: 1,}).sort({ts: -1}).limit(1)
{ "ts" : Timestamp(1542563322, 1) }
rs0:PRIMARY>

2. On one of the members of your live replica set, dump the oplog collection using mongodump:

[root@ip-172-31-93-199 mongodump]# mongodump -u admin -p XXXXXX --authenticationDatabase admin -d local -c oplog.rs -o oplogdump
2018-11-18T17:55:20.593+0000 writing local.oplog.rs to
2018-11-18T17:55:20.614+0000 done dumping local.oplog.rs (323 documents)
[root@ip-172-31-93-199 mongodump]# cd oplogdump/
[root@ip-172-31-93-199 oplogdump]# ls
local
[root@ip-172-31-93-199 oplogdump]# cd local/
[root@ip-172-31-93-199 local]# ls
oplog.rs.bson  oplog.rs.metadata.json

It is not necessary Oplog will have a record created exact on the same time, it might create a record earlier or later to this time. And obviously that will happen most of the time. In that scenario, you must find an epoch value just less than the required epoch time and you will restore up to that timestamp. Your goal is to find “”ts”:{“timestamp”:{“t”:x,”i”:y}}” entry and note its values from Oplogs. X is the epoch value we are interested in and Y is an incrementing ordinal for operations within a given second. In case you find more than 1 value of Y for the same value of X, you should use the largest Y value.

3. Connect to the server using the mongo shell, find the bad operation in the oplog and make a note of the time_t:ordinal (timestamp) values. Alternatively, use bsondump to manually inspect the BSON dump of the oplog.

[root@ip-172-31-93-199 local]# bsondump oplog.rs.bson > oplogrecovery.json
2018-11-18T18:05:49.359+0000    323 objects found
[root@ip-172-31-93-199 local]# ls
oplogrecovery.json  oplog.rs.bson  oplog.rs.metadata.json

4.  Find the timestamp of the bogus transaction, which represents the point until you want to replay the oplog:

[root@ip-172-31-93-199 local]# tail -n300 oplogrecovery.json
...
{"ts":{"$timestamp":{"t":1542562302,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"6091652354277864417"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:31:42.796Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562312,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7661599134073288470"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:31:52.797Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562322,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"-2999680384224794684"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:32:02.797Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562328,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"-7814033733276863336"},"v":2,"op":"c","ns":"testdata.$cmd","ui":{"$binary":"T+Y32KP0R1mHznHqnQP2Cg==","$type":"04"},"wall":{"$date":"2018-11-18T17:32:08.450Z"},"o":{"drop":"birthday"}}
{"ts":{"$timestamp":{"t":1542562342,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"8689417822344734569"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:32:22.797Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562352,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7595615792183348688"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:32:32.798Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562362,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"1667414361006145572"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:32:42.798Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562372,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"-4034500875267832055"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:32:52.798Z"},"o":{"msg":"periodic noop"}}
{"ts":{"$timestamp":{"t":1542562382,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7657953432518951006"},"v":2,"op":"n","ns":"","wall":{"$date":"2018-11-18T17:33:02.799Z"},"o":{"msg":"periodic noop"}}


recovery

In this case drop happened at time stamp {“t”:1542562328,”i”:1}  and we need to restore till {“t”:1542562322,”i”:1}

5. Start a new standalone mongod instance to rebuild the server and verify the data before applying it to production. If you use mongorestore to restore a backup.Start a new mongod process, and restore the last known good backup using mongorestore :

[root@ip-172-31-93-199 mongodump]# mongorestore -uadmin -p XXXXX --authenticationDatabase admin  -d testdata /mongodump/testdata/birthday.bson
2018-11-18T18:19:33.339+0000    checking for collection data in /mongodump/testdata/birthday.bson
2018-11-18T18:19:33.360+0000    reading metadata for testdata.birthday from /mongodump/testdata/birthday.metadata.json
2018-11-18T18:19:33.390+0000    restoring testdata.birthday from /mongodump/testdata/birthday.bson
2018-11-18T18:19:33.469+0000    no indexes to restore
2018-11-18T18:19:33.469+0000    finished restoring testdata.birthday (2000 documents)
2018-11-18T18:19:33.469+0000    done

Verify the data after dump restoration

s0:PRIMARY> use testdata
switched to db testdata
rs0:PRIMARY> show collections
birthday
test
testcoll
rs0:PRIMARY> db.birthday.find().count()
2000

mongorestore has restore 2000 documents out of 2001 data count

6.  Now you’re ready to replay the oplog using — oplogLimit to set the delimiter:

[root@ip-172-31-93-199 local]# mongorestore -u admin -p XXXXX --authenticationDatabase admin  --oplogReplay --oplogLimit  1542562322:1 /mongodump/oplogdump/local/oplog.rs.bson
2018-11-18T18:28:29.182+0000    checking for collection data in /mongodump/oplogdump/local/oplog.rs.bson
2018-11-18T18:28:29.182+0000    replaying oplog
2018-11-18T18:28:29.207+0000    done

7. Check that all documents are now back in the collection and data has been correctly restored.

rs0:PRIMARY> use testdata
switched to db testdata
rs0:PRIMARY> db.birthday.find().count()
2001

restored

8. Depending on the procedure used in the previous steps, restore the dump or the data files into the live/production replica set.

References:

https://www.percona.com/blog/2016/09/20/mongodb-point-in-time-backups-made-easy/

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Powered by WordPress.com.

Up ↑

%d bloggers like this: