MongoDB for DBA's 4/7. Replication. Homeworks

Homework 4.1

In this chapter’s homework we will create a replica set and add some data to it.

1. Unpack replication.js from the Download Handout zip file.

2. We will create a three member replica set. Pick a root working directory to work in. Go to that directory in a console window.

Given we will have three members in the set, and three mongod processes, create three data directories:

mkdir 1
mkdir 2
mkdir 3

3. We will now start a single mongod as a standalone server. Given we will have three mongod processes on our single test server, we will explicitly specify the port numbers (this wouldn’t be necessary if we had three real machines or three virtual machines). We’ll also use the --smallfiles parameter and --oplogSize so the files are small given we have a lot of server processes running on our test PC.

$ # starting as a standalone server for problem 1:
$ mongod --dbpath 1 --port 27001 --smallfiles --oplogSize 50

Note: for all mongod startups in the homework this chapter, you can optionally use --logPath, --logappend, and --fork. Or, since this is just an exercise on a local PC, you could simply have a separate terminal window for all and forgo those settings. Run “mongod --help” for more info on those.

mongod --dbpath 1 --port 27001 --smallfiles --logpath log.1 --logappend --fork --oplogSize 50
mongod --dbpath 2 --port 27002 --smallfiles --logpath log.2 --logappend --fork --oplogSize 50
mongod --dbpath 3 --port 27003 --smallfiles --logpath log.3 --logappend --fork --oplogSize 50

To know pid mongo processes:

ps -Aef | grep mongod

To kill mongod processes

kill pid

4. In a separate terminal window (cmd.exe on Windows), run the mongo shell with the replication.js file:

mongo --port 27001 --shell replication.js

Then run in the shell:

> homework.init()

This will load a small amount of test data into the database.

Now run:

 > homework.a()

and enter the result. This will simply confirm all the above happened ok.

result: 5001

Homework 4.2

Now convert the mongod instance (the one in the problem 4.1 above, which uses “--dbpath 1”) to a single server replica set. To do this, you’ll need to stop the mongod (NOT the mongo shell instance) and restart it with “--replSet” on its command line. Give the set any name you like.

ps -Aef | grep mongod

mongodb   1009     1  0 09:25 ?        00:00:05 /usr/bin/mongod --config /etc/mongod.conf
user  3260  2009  0 09:42 ?        00:00:02 mongod --dbpath 1 --port 27001 --smallfiles --logpath log.1 --logappend --fork --oplogSize 50
user  3272  2009  0 09:43 ?        00:00:01 mongod --dbpath 2 --port 27002 --smallfiles --logpath log.2 --logappend --fork --oplogSize 50
user  3284  2009  0 09:43 ?        00:00:01 mongod --dbpath 3 --port 27003 --smallfiles --logpath log.3 --logappend --fork --oplogSize 50

user  3360  3221  0 09:48 pts/0    00:00:00 grep --color=auto mongod

To kill mongod processes

kill 3260

mongod --replSet abc --dbpath 1 --port 27001 --smallfiles --logpath log.1 --logappend --fork --oplogSize 50

Then go to the mongo shell. Once there, run

> rs.initiate()

> rs.initiate()

{

"info2" : "no configuration explicitly specified -- making one",

"me" : "SERVER:27001",

"info" : "Config now saved locally.  Should come online in about a minute.",

"ok" : 1

}

rs.status()

{

"set" : "abc",

"date" : ISODate("2015-01-30T08:55:02Z"),

"myState" : 1,

"members" : [

{

"_id" : 0,

"name" : "SERVER:27001",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 364,

"optime" : Timestamp(1422607970, 1),

"optimeDate" : ISODate("2015-01-30T08:52:50Z"),

"electionTime" : Timestamp(1422607970, 2),

"electionDate" : ISODate("2015-01-30T08:52:50Z"),

"self" : true

}

],

"ok" : 1

}

abc:PRIMARY> rs.conf()

{

"_id" : "abc",

"version" : 1,

"members" : [

{

"_id" : 0,

"host" : "SERVER:27001"

}

]

}

When you first ran homework.init(), we loaded some data into the mongod. You should see it in the replication database. You can confirm with:

> use replication
> db.foo.find()

Once done with that, run

> homework.b()

in the mongo shell and enter that result below.

result: 5002

Homework 4.3

Now add two more members to the set. Use the 2/ and 3/ directories we created in homework 4.1. Run those two mongod’s on ports 27002 and 27003 respectively (the exact numbers could be different).

Remember to use the same replica set name as you used for the first member.

kill 3272 3484

mongod --replSet abc --dbpath 2 --port 27002 --smallfiles --logpath log.2 --logappend --fork --oplogSize 50

mongod --replSet abc --dbpath 3 --port 27003 --smallfiles --logpath log.3 --logappend --fork --oplogSize 50

You will need to add these two new members to your replica set, which will initially have only one member. In the shell running on the first member, you can see your replica set status with

> rs.status()

Initially it will have just that first member. Connecting to the other members will involve using

rs.add()

. For example,

> rs.add("localhost:27002")

You'll know it's added when you see an

{ "ok" : 1 }

document.

Your machine may or may not be OK with 'localhost'. If it isn't, try using the name in the "members.name" field in the document you get by calling rs.status() (but remember to use the correct port!).

abc:PRIMARY> rs.conf()

{

"_id" : "abc",

"version" : 3,

"members" : [

{

"_id" : 0,

"host" : "SERVER:27001"

},

{

"_id" : 1,

"host" : "SERVER:27002"

},

{

"_id" : 2,

"host" : "SERVER:27003"

}

]

}

Once a secondary has spun up, you can connect to it with a new mongo shell instance.

mongo --port 27002

Use

rs.slaveOk()

to let the shell know you're OK with (potentially) stale data, and run some queries. You can also insert data on your primary and then read it out on your secondary. Once the servers have sync'd with the primary and are caught up, run (on your primary):

> homework.c()

and enter the result below.

result: 5

Homework 4.4

We will now remove the first member (@ port 27001) from the set.

As a first step to doing this we will shut it down. (Given the rest of the set can maintain a majority, we can still do a majority reconfiguration if it is down.)

We could simply terminate its mongod process, but if we use the replSetStepDown command, the failover may be faster. That is a good practice, though not essential. Connect to member 1 (port 27001) in the shell and run:

> rs.stepDown()

Then cleanly terminate the mongod process for member 1.

abc:SECONDARY> rs.status()

{

"set" : "abc",

"date" : ISODate("2015-01-30T09:42:11Z"),

"myState" : 2,

"syncingTo" : "SERVER:27003",

"members" : [

{

"_id" : 0,

"name" : "SERVER:27001",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 621,

"optime" : Timestamp(1422610708, 149),

"optimeDate" : ISODate("2015-01-30T09:38:28Z"),

"infoMessage" : "syncing to: SERVER:27003",

"self" : true

},

{

"_id" : 1,

"name" : "SERVER:27002",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 398,

"optime" : Timestamp(1422610708, 149),

"optimeDate" : ISODate("2015-01-30T09:38:28Z"),

"lastHeartbeat" : ISODate("2015-01-30T09:42:10Z"),

"lastHeartbeatRecv" : ISODate("2015-01-30T09:42:11Z"),

"pingMs" : 0,

"lastHeartbeatMessage" : "syncing to: SERVER:27001",

"syncingTo" : "SERVER:27001"

},

{

"_id" : 2,

"name" : "SERVER:27003",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 394,

"optime" : Timestamp(1422610708, 149),

"optimeDate" : ISODate("2015-01-30T09:38:28Z"),

"lastHeartbeat" : ISODate("2015-01-30T09:42:11Z"),

"lastHeartbeatRecv" : ISODate("2015-01-30T09:42:11Z"),

"pingMs" : 0,

"electionTime" : Timestamp(1422610890, 1),

"electionDate" : ISODate("2015-01-30T09:41:30Z")

}

],

"ok" : 1

}

Next, go to the new primary of the set. You will probably need to connect with the mongo shell, which you'll want to run with '--shell replication.js' since we'll be getting the homework solution from there. 

mongo --port 27003 --shell replication.js

Once you are connected, run rs.status() to check that things are as you expect. Then reconfigure to remove member 1.

Tip: You can either use rs.reconfig() with your new configuration that does not contain the first member, or rs.remove(), specifying the host:port of the server you wish to remove as a string for the input.

rs.remove("SERVER:27001")

When done, run

> homework.d()

and enter the result.

Tip: If you ran the shell without replication.js on the command line, restart the shell with it.

 result: 6

Homework 4.5

Note our replica set now has an even number of members, and that is not a best practice. However, to keep the homework from getting too long we’ll leave it at that for now, and instead do one more exercise below involving the oplog.

To get the right answer on this problem, you must perform the homework questions in order. Otherwise, your oplog may look different than we expect.

Go to the secondary in the replica set. The shell should say SECONDARY at the prompt if you've done everything correctly.

mongo --port 27002 --shell replication.js

Switch to the local database and then look at the oplog:

> db.oplog.rs.find()

If you get a blank result, you are not on the right database.

Note: as the local database doesn’t replicate, it will let you query it without entering “rs.slaveOk()” first.

Next look at the stats on the oplog to get a feel for its size:

> db.oplog.rs.stats()

What result does this expression give when evaluated?

db.oplog.rs.find().sort({$natural:1}).limit(1).next().o.msg[0]

 result: R