MongoDB course for developers. unit 6/8. Application Engineering. Homeworks

Homework 6.1

Which of the following statements are true about MongoDB replication. Check all that apply.

The minimum sensible number of voting nodes to a replica set is three.MongoDB replication is synchronous.The Mongo shell is capable of attaching to a replica set and automatically failing over.By default, using the new MongoClient connection class, w=1 and j=1.The oplog utilizes a capped collection.

Homework 6.2

Let's suppose you have a five member replica set and want to assure that writes are committed to the journal and are acknowledged by at least 3 nodes before you proceed forward. What would be the appropriate settings for w and j?

w=1, j=1w="majority", j=1w=3, j=0w=5, j=1w=1,j=3

Homework 6.3

Which of the following statements are true about choosing and using a shard key:

The shard key must be uniqueThere must be a index on the collection that starts with the shard key.Mongo can not enforce unique indexes on a sharded collection other than the shard key itself.Any update that does not contain the shard key will be sent to all shards.You can change the shard key on a collection if you desire.

Homework 6.4

You have a sharded system with three shards and have sharded the collections "grades" in the "test" database across those shards. The output of sh.status() when connected to mongos looks like this:

mongos> sh.status()
--- Sharding Status --- 
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
 {  "_id" : "s0",  "host" : "s0/localhost:37017,localhost:37018,localhost:37019" }
 {  "_id" : "s1",  "host" : "s1/localhost:47017,localhost:47018,localhost:47019" }
 {  "_id" : "s2",  "host" : "s2/localhost:57017,localhost:57018,localhost:57019" }
  databases:
 {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
 {  "_id" : "test",  "partitioned" : true,  "primary" : "s0" }
  test.grades chunks:
    s1 4
    s0 4
    s2 4
   { "student_id" : { $minKey : 1 } } -->> { "student_id" : 0 } on : s1 Timestamp(12000, 0) 
   { "student_id" : 0 } -->> { "student_id" : 2640 } on : s0 Timestamp(11000, 1) 
   { "student_id" : 2640 } -->> { "student_id" : 91918 } on : s1 Timestamp(10000, 1) 
   { "student_id" : 91918 } -->> { "student_id" : 176201 } on : s0 Timestamp(4000, 2) 
   { "student_id" : 176201 } -->> { "student_id" : 256639 } on : s2 Timestamp(12000, 1) 
   { "student_id" : 256639 } -->> { "student_id" : 344351 } on : s2 Timestamp(6000, 2) 
   { "student_id" : 344351 } -->> { "student_id" : 424983 } on : s0 Timestamp(7000, 2) 
   { "student_id" : 424983 } -->> { "student_id" : 509266 } on : s1 Timestamp(8000, 2) 
   { "student_id" : 509266 } -->> { "student_id" : 596849 } on : s1 Timestamp(9000, 2) 
   { "student_id" : 596849 } -->> { "student_id" : 772260 } on : s0 Timestamp(10000, 2) 
   { "student_id" : 772260 } -->> { "student_id" : 945802 } on : s2 Timestamp(11000, 2) 
   { "student_id" : 945802 } -->> { "student_id" : { $maxKey : 1 } } on : s2 Timestamp(11000, 3) 

If you ran the query

use test
db.grades.find({'student_id':530289})

Which shards would be involved in answering the query?

s0,s1 and s2s0s1s2

Homework 6.5

Create three directories for the three mongod processes. On unix, this could be done as follows:

mkdir -p /data/rs1 /data/rs2 /data/rs3

Now start three mongo instances as follows. Note that are three commands. The browser is probably wrapping them visually.

./mongod --replSet m101 --logpath "1.log" --dbpath /data/rs1 --port 27017 --smallfiles --fork

./mongod --replSet m101 --logpath "2.log" --dbpath /data/rs2 --port 27018 --smallfiles --fork

./mongod --replSet m101 --logpath "3.log" --dbpath /data/rs3 --port 27019 --smallfiles --fork

Now connect to a mongo shell and make sure it comes up

./mongo --port 27017

Now you will create the replica set. Type the following commands into the mongo shell:

config = { _id: "m101", members:[
          { _id : 0, host : "localhost:27017"},
          { _id : 1, host : "localhost:27018"},
          { _id : 2, host : "localhost:27019"} ]
};
rs.initiate(config);

At this point, the replica set should be coming up. You can type

rs.status()

to see the state of replication. 

m101:PRIMARY> rs.status()

{

"set" : "m101",

"date" : ISODate("2014-12-04T18:50:17Z"),

"myState" : 1,

"members" : [

{

"_id" : 0,

"name" : "localhost:27017",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 234,

"optime" : Timestamp(1417718826, 1),

"optimeDate" : ISODate("2014-12-04T18:47:06Z"),

"electionTime" : Timestamp(1417718835, 1),

"electionDate" : ISODate("2014-12-04T18:47:15Z"),

"self" : true

},

{

"_id" : 1,

"name" : "localhost:27018",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 190,

"optime" : Timestamp(1417718826, 1),

"optimeDate" : ISODate("2014-12-04T18:47:06Z"),

"lastHeartbeat" : ISODate("2014-12-04T18:50:17Z"),

"lastHeartbeatRecv" : ISODate("2014-12-04T18:50:17Z"),

"pingMs" : 0,

"syncingTo" : "localhost:27017"

},

{

"_id" : 2,

"name" : "localhost:27019",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 190,

"optime" : Timestamp(1417718826, 1),

"optimeDate" : ISODate("2014-12-04T18:47:06Z"),

"lastHeartbeat" : ISODate("2014-12-04T18:50:17Z"),

"lastHeartbeatRecv" : ISODate("2014-12-04T18:50:17Z"),

"pingMs" : 0,

"syncingTo" : "localhost:27017"

}

],

"ok" : 1

}

MongoDB course for developers. unit 6/8. Application Engineering

Write Concern

It's how concern are your writes complete before you get the responses back ?

All of this is controlled by the drivers. The driver and mongo shell will execute for you the function getLastError after the write operation. In mongo the operations are not acknowledge and requires a second call to getLastError function.

Every single time yo do an insert o update (a single operation), the function getLastError is called in order to get a possible error. If you use a driver to access mongodb database you can decide if these drivers will call or not the function getLastError.

The function getLastError can have two parameters:

• w: (write) if it's equal to 1, it determines weather or not you want to wait for the write operation to be acknowledge.
• j: (journal = log to disk with the operations with the data) if it's equal to 1, getLastError waits until the journal commits to disk

Values of parameters:

• w = 0, j = 0 : fire and forget 
• w = 1, j = 0 : wait for a simple acknowlegement from mongo that receives the write. BY DEFAULT
• w = 0 or 1 , j = 1 : wait for the write commits the journal. TO SURE THAT YOU ARRIVE TO DISK BEFORE YOU GO ON 

Provided you assume that the disk is persistent, what are the w and j settings required to guarantee that an insert or update has been written all the way to disk.

w=0, j=0 w=1, j=1 w=2, j=0 w=1, j=0

Network Errors

We can have network errors that provoque a write or insert cannot arrive to disk even though the options of write concern will set to true (w=1, j=1). We can never complete sure what exactly happened in the transaction. In order to solve this situations you can use a 'try exception' to get the error and offer a solution.

What are the reasons why an application may receive an error back even if the write was successful. Check all that apply.

The network TCP network connection between the application and the server was reset between the time of the write and the time of the getLastError call.The MongoDB server terminates between the write and the getLastError call.The network fails between the time of the write and the time of the getLastError callThe write violates a primary key constraint on the collection and must be rolled back.

The Pymongo Driver

The api web site is api.mongodb.org . This is the main directory of all the drivers to use with mongodb as a developer.

The current driver will always be at api.mongodb.org/python/current

The most important takeaways are that you should be using pymongo.MongoClient() to connect to a standalone server, or if you're connecting to a replica set, an even better option is pymongo.MongoReplicaSetClient().

Which of the following are valid, supported ways to connect to a server with pymongo?

pymongo.Connection()

pymongo.MongoClient()

pymongo.MongoReplicaSetClient()

pymongo.MongoShardedClusterClient()

Introduction to Replication

Replication offers fault tolerance in order to the system continues working when a node goes down or we have an accident like a fire.

The solution of mongo is building a replica set. A replica set is a group of nodes with mongod that works mirroring each other the data. The is one primary node and the others are secondaries dynamically.

The operation is that your application and its drivers stay connected to the primary node, and will write to the primary (you can only write to the primary). If the primary goes down, the reamining nodes will perfom an election to elect a new primary having a strict majority of the original nodes.

The minimun number of nodes to buid a replica set is three and you can have an arbiter node to decide which one will be primary in case of a tie.

What is the minimum original number of nodes needed to assure the election of a new Primary if a node goes down?

1 2 3 5

Replica Set Elections

The Replica set is totally transparent for applications that will continue working without any break.

Types of Replica set nodes:

• Regular: has the data and it's the most normal type of node. It can be a primary or secondary
• Arbiter: It can be a regular node. It's used for voting purposes. If you have even number of replica set nodes you need to make sure that there's an arbiter node in order to have a strict majority to elect a node as primary
• Delayed/Regular: It offers the possibility to delay behind other nodes to recover data in a fast way. It cannot be a primary but can participate voting the election. Its priority is set to zero.
• Hidden: it's often used for analytics.

Setting the priority to zero, the node will not elected as a primary. It cannot be a primary but can participate voting the election. Its priority is set to zero.

 

Wihich types of nodes can participate in elections of a new primary?

Regular replica set membersHidden MembersArbitersLawyers

Write Consistency

The writes are sent to the primary node but the reads can send to any node (by default is set to the primary node to have a good consistency) but keep in mind tha the lag between any two nodes is no guaranteed becase the replication is asynchronous and data read cannot exist at the time of reading.

 

During the time when failover is occurring, can writes successfully complete?

YesNo

Creating a Replica Set

In order to create a Replica set, we have to create and next initizalize their nodes (in this case the three nodes are in one server with differents ports. In other case it will be necessary specify th target host and port for each node):

1) Create the nodes of Replica set:
mongod --port 27017 --dbpath "/var/lib/mongodb/data/rs1" --replSet group1 --logpath "log_1.log" --oplogSize 200 --fork --smallfiles

mongod --port 27018 --dbpath "/var/lib/mongodb/data/rs2" --replSet group1 --logpath "log_2.log" --oplogSize 200 --fork --smallfiles

mongod --port 27019 --dbpath "/var/lib/mongodb/data/rs3" --replSet group1 --logpath "log_3.log" --oplogSize 200 --fork --smallfiles

2) Initialise the Replica set creating a variable within the configuration and loading it from the shell.

config = {
    "_id" : "group1",
    "members" : [
    //node 1
     { "_id" : 1 , "host" : "localhost:27017"}
    //node 2
    ,{ "_id" : 2 , "host" : "localhost:27018"}
    //node 3
    ,{ "_id" : 3 , "host" : "localhost:27019"}
    ]
};

rs.initiate(config);

To know the status of replica set:

rs.status();

To allow readings from an slave node it's necessary to execute in the secondary node:

rs.slaveOk()
 

To show which node is the master:

rs.isMaster()

To force the current replica set member to step down as primary and then attempt to avoid election as primary for the designated number of seconds (60 second by default). Produces an error if the current member is not the primary.

rs.stepDown()

To show help of replica set options:

rs.help()

Which command, when issued from the mongo shell, will allow you to read from a secondary?

db.isMaster()db.adminCommand({'readPreference':'Secondary'})rs.setStatus("Primary")rs.slaveOk()

Replica Set Internals

In the video how long did it take to elect a new primary?

About three secondsAbove 10 secondsAbout 30 secondsAbout a minute

 

Failover and Rollback

What happens if a node comes back up as a secondary after a period of being offline and the oplog has looped on the primary?

The entire dataset will be copied from the primary

A rollback will occur

The new node stays offline (does not re-join the replica set)

The new node begins to calculate Pi to a large number of decimal places

Connecting to a Replica Set from Pymongo

c = pymongo.MongoClient(host=

[

                 "mongodb://localhost:27017",
                 "mongodb://localhost:27018",
                 "mongodb://localhost:27019"

],
                 replicaSet="rs1",
                 w=1, j=True)

or

 

c = pymongo.MongoClient(host=

[

                 "mongodb://localhost:27017"

],
                 replicaSet="rs1",
                 w=1, j=True)

 

If you leave a replica set node out of the seedlist within the driver, what will happen?

The missing node will not be used by the application.The missing node will be discovered as long as you list at least one valid node.This missing node will be used for reads, but not for writes.The missing node will be used for writes, but not for reads.

What happens when the failover occurs

When the election is happening you can't complete writes or reads because yo don't have a primary to go to.

What will happen if the following statement is executed in Python during a primary election?

db.test.insert({'x':1})

Insert will fail, program will continue

Insert will succeed to new primary, because failover is automatic within MongoDB.

Insert will fail, program will terminate

Insert will succeed, program will terminate

 

Detecting Failover

If you catch exceptions during failover, are you guaranteed to have your writes succeed?

Yes No

 

Proper Handling of Failover

Example in python of code to use in replica set to hand a failover:

 

def writesome():
    # let's do some inserting
    for i in range(0,1000000):
        for retries in range(0,3):

 

        # 'doc' is here because if a exception is produced the test.insert(doc) 

        # will insert a document with a different id and it will not provoke a  

        # duplicateKeyError

            doc = {'i':i}           
        try:
                test.insert(doc)
                print "Inserted " + str(i)
                break
            except pymongo.errors.DuplicateKeyError:
                print "Duplicate key error"
                break
            except:
                print sys.exc_info()[0]
                print "Retrying..."
                time.sleep(5)
        time.sleep(.5)

 

If this code guaranteed to get the write done if failover occurs:

doc = {'i':i}
        for retries in range(0,3):

            try:
                test.insert(doc)
                print "Inserted " + str(i)
                break
            except pymongo.errors.DuplicateKeyError:
                print "Duplicate key error"
                break
            except:
                print sys.exc_info()[0]
                print "Retrying..."
                time.sleep(5)

Yes No 

Write concern revisited

Options:

• w = 1: it will wait until the primary node makes the write
• w = 2: it will wait until two nodes make the write (if we have three nodes)
• w = 3: it will wait until three nodes make the write (if we have three nodes)
• j = 1: it will wait until the primary node 
• wtimeout (seconds): how long you wiloing to wait for the writes to be acknowledged by the secondaries. (it can be set in the drivers)

 There are three differents places where can be set this options:

1. on the connection
2. on the connections inside the driver
3. in the configuration itself of the Replica set you can set default values

 If you set w=1 and j=1, is it possible to wind up rolling back a committed write to the primary on failover?

Yes No

Read preferences

You can specify read preference to read specific secondary nodes.

Options in Pymongo:

1. always read on the primary
2. always read on the secondary. If there isn't secondary the reads cannot do
3. secondary preference and if there isn't it will read from a primary
4. primary preference and if there isn't it will read from a secondary
5. the nearest node
6. by tagging. You can assing tags to nodes in order to name them

 You can configure your applications via the drivers to read from secondary nodes within a replica set. What are the reasons that you might not want to do that? Check all that apply.

If your write traffic is significantly greater than your read traffic, you may overwhelm the secondary, which must process all the writes as well as the reads. Replication lag can result.

You may not read what you previously wrote to MongoDB.

  If the secondary hardware has insufficient memory to keep the read working set in memory, directing reads to it will likely slow it down.

Reading from a secondary prevents it from being promoted to primary.

Implications of replication

1. Seed list to ensure that an election will done when the primary goes down
2. Write concern: the idea of waiting for some number of nodes to acknowlege the writes through to w parameter, the j parameter which lets it wait or not for the primary node to commit that write to disk. An wtimeout parameter, which is how long you are going to wait to see that your write replicated to other members of the replica set. 
3. Read preferences
4. Errors can have: errors can always happen because of transient situations like failover occuring, or they can happen because there are network errors that occurs o errors in terms of violating the unique key constraints

To create a robust application it is necessary to check for exceptions of read and write operations to database in order to make sure that if anything comes up the application will know it. It is necessary to make sure we understand the application of what data has been committed and whata data is durable in the application.

1. Seed list to ensure that an election will done when the primary goes down
2. Write concern: the idea of waiting for some number of nodes to ack

If you set w=4 on a connection and there are only three nodes in the replica set, how long will you wait in PyMongo for a response from an insert if you don't set a timeout?

Comes back immediately

About a minute

About 10 seconds

More than five minutes

 

Introduction to Sharding

 This is an approach to horizontal scalability. Every shard node can have their replica set because of this we have a lot of hosts involved.

In order to distribute the data, mongo uses a router named 'mongos' that's going to take care of the distribution. It's going to keep some sort of connection pool or knowledge of all the different hosts, and it's going to route them properly.

shard_key: something that is going to determine, it's some part of the document itself  (the _id of the document)

 

Once you make the decission of what kind of shard key to use, mongo will then break the collection into chunks and decide what shard each of the chunks lives on a range-based way, and then any query that you make, which now has to be routed to a Mongo OS will then go to the appropiate shards to answer your query.

 

If the shard key is not include in a find operation and there are 3 shards, each one a replica set with 3 nodes, how many nodes will see the find operation?

1396

Building a Sharded Environment

If you want to build a production system with two shards, each one a replica set with three nodes, how may mongod processes must you start?

2679

2 shards has 6 nodes. 3 config nodes

Implications of Sharding

 Some things to remember in a shard environment:

1. Every document needs to include the shard key
2. The shard key is immutable: yo cannot change the shard key inside the document
3. it needs an index that starts with the shard key but it cannot be a multiple index
4. when do an update, it's necessary to specify the shard key or specify that multi is true
5. no shard key means scatter gather operation, which could be expensive
6. you can't have a unique key, no unique index, unless it's also part of the shard key

Suppose you wanted to shard the zip code collection after importing it. You want to shard on zip code. What index would be required to allow MongoDB to shard on zip code?

An index on zip or a non-multi-key index that starts with zip.No index is required to use zip as the shard key.A unique index on the zip code.Any index that that includes the zip code.

Sharding + Replication

Suppose you want to run multiple mongos routers for redundancy. What level of the stack will assure that you can failover to a different mongos from within your application?

mongodmongosdriverssharding config servers

Choosing a Shard Key

1. Sufficient cardinality: in order to mongo can distribute the documents in shards
2. avoid hotspots in writes: monotically increasing: the shard key will not provoque that all the writes will go to an specific shard

You are building a facebook competitor called footbook that will be a mobile social network of feet. You have decided that your primary data structure for posts to the wall will look like this:

{'username':'toeguy',
     'posttime':ISODate("2012-12-02T23:12:23Z"),
     "randomthought": "I am looking at my feet right now",
     'visible_to':['friends','family', 'walkers']}

Thinking about the tradeoffs of shard key selection, select the true statements below.

Choosing posttime as the shard key will cause hotspotting as time progresses.Choosing username as the shard key will distribute posts to the wall well across the shards.Choosing visible_to as a shard key is illegal.Choosing posttime as the shard key suffers from low cardinality.