Buscar

miércoles, 14 de enero de 2015

MongoDB for DBA's 2/7. Crud

Chapter 2. CRUD: Creating, reading and updating data

Insert operation

show dbs
use database
db --> show current database
show collections --> show the current database collections
db.sample.insert ( { a : 1 }) --> to insert a document
db.sample.find( )  --> to query documents
db.sample.find( ).pretty()  --> to query documents in compact format

a documents can have differents schemes.

var aa = db.sample.find( ).toArray()   --> put the result data in an array and assign to a variable aa like javascript

db.getLastError()  -->  to force the acknowlodge of the insert operation


Insert the document
{ x : 3 , y : 4 }
into the temperature collection for the current database. 

The collection will initially be empty, but you can (and should) check for your document after insertion. The shell will automatically add an _id field to the inserted document. You should only insert the one document.



> db.temperature.insert({x:3,y:4})

> WriteResult({ "nInserted" : 1 })
> db.temperature.find()
{ "_id" : ObjectId("54b5835f132c1f084b02ab21"), "y" : 4, "x" : 3 }

Update operation

you can update one o more documents in a collection. 
db.colection.update (<<where>>, 
                                 <<doc or partial update  expression in JSON or BSON format >> 
                                 , <upsert>  --> optional
                                 , <multi>  -->  optional
                                 )
There are two kinds of updates:

full document update replacement

partial update: only change a single field

<upsert> : update or insert if not present
<multi>  : many documents not only one
t = db.sample
t.update( { "_id" : 100}, { "a" : 100 } ) --> the id field must not change instead of we get an error

myobj = t.findOne()
{ "_id" : 190, x : "hello" }
myobj.y = 123

{ "_id" : 190, x : "hello", y : 123 }
t.update( { "_id" : myobj._id}, myobj )







Check all that are true about the _id field:

Document growth / relocation


In v2.6 and higher, by default --> powerOf2sizes = on

Having powerOf2sizes on, the smallest documents you insert will be allocated in 32 byte allocation units. Then, it it's greater than that size, the next size will be 64 bytes, 128 bytes, etc. until to 16Mbytes limit. After 4MB it actually adds only 1MB at a time since we will getting closed to the upper limit .

The reason for this sizing strategy is to avoid heap fragmentation or deleted space fragmentation.
If you have a collection where you really needed to optimize, and you wanted a very specific allocation unit size you could turn this off.

What operations could cause an existing document to be moved on disk? Check all that apply.

Save() command


It's a shell helper function, it's not a mongoDB server operation,

myobj = t.findOne()
{ "_id" : 190, x : "hello" }
myobj.y = 897

t.save(myobj)

{ "_id" : 190, x : "hello", y : 897 }

What happens if you try to use db.collection.save(document) if the inserted document has no _id?

Partial Updates & Documents

The size limit for each document is 16MBytes in order not to saturate the ethernet connection in  high documents like that 100MB.

We can use operators like:

  1. $set : to set a new value
  2. $push: to add somethig to an array. If it's not exist, it will be created
  3. $addToSet: to add to an array if not already present
  4. $pop : to remove from an array
  5. $unset: to delete a field
  6. etc ...

The Mongo Web Shell has been initialized with one collection, cars, with one document preexisting:
{ "_id" : 100, "name" : "GTO", "year" : 1969, "color" : "red" }
Set the available field to 1. 

You can verify the initial state of the collection, and verify your answer before you submit. There should be only one document in the collection. 

> db.cars.find()
{ "_id" : 100, "color" : "red", "name" : "GTO", "year" : 1969 }
> db.cars.update({ "_id":100}, { "$set":{"available":1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.cars.find()
{
    "_id" : 100,
    "color" : "red",
    "available" : 1,
    "name" : "GTO",
    "year" : 1969
}


Removing Documents

db.<< collection name>>.remove( << expression >>)

This is a multidocument operation.

db.test.remove( { "_id" : 100 } )

db.test.remove( { "x" : /hello/ })  --> with regular expressions


We have initialized documents in the users collection of the form:
{
 _id : ObjectId("50897dbb9b96971d287202a9"),
 name : "Jane",
 likes : [ "tennis", "golf" ],
 registered : false,
 addr : {
   city : "Lyon",
   country : "France" 
 } 
}
Delete all documents in the collection where city is "Lyon" and registered is false. 

You can check the initial state of the collection, and verify your answer before you submit. 

> db.users.count()
300
> db.users.remove({ "addr.city" : "Lyon", "registered" : false})
WriteResult({ "nRemoved" : 33 })
> db.users.count()
267


Multi Update

db.collection.update( query_document , update_document , [ options_document ] )

where optional options_document has any one or more of the following optional parameters:

upsert : true/false,
multi : true/false,
writeConcern: document

by default --> upsert : false, multi : false

Which of the following are disadvantages to setting multi=false (as it is by default)?
Upsert

updates or inserts if not present


We have initialized documents in the users collection, one of which is the following:
{
 _id : "Jane",
 likes : [ "tennis", "golf" ],
 registered : false,
 addr : {
   city : "Lyon",
   country : "France" 
 } 
}
In the shell, add that this user likes "football". You should not need to pass "tennis" or "golf" to theupdate query at all. 

You can look at the initial state of the collection, and verify your answer before you submit. 
> db.users.count()
11
> db.users.find( { "likes" : "football" })
> db.users.update( {}, { "$addToSet" :{ "likes" : "football" }}, { "upsert":true,"multi":true})
WriteResult({ "nMatched" : 11, "nUpserted" : 0, "nModified" : 11 })

Wire Protocol


What are the basic building blocks of the wire protocol? Check all that apply.

Which operation is overloaded in the wire protocol to handle commands?

Bulk() write Operations & Methods

There are two basic forms:

  1. ordered: With an ordered list of operations, MongoDB executes the operations serially. If an error occurs during the processing of one of the write operations, MongoDB will return without processing any remaining write operations in the list.
  2. unordered: With an unordered list of operations, MongoDB can execute the operations in parallel. If an error occurs during the processing of one of the write operations, MongoDB will continue to process remaining write operations in the list.
To use the Bulk() methods:

  1. Initialize a list of operations using either db.collection.initializeUnorderedBulkOp() ordb.collection.initializeOrderedBulkOp().
  2. Add write operations to the list using the following methods:
    • Bulk.insert()
    • Bulk.find()
    • Bulk.find.upsert()
    • Bulk.find.update()
    • Bulk.find.updateOne()
    • Bulk.find.replaceOne()
    • Bulk.find.remove()
    • Bulk.find.removeOne()
  3. To execute the list of operations, use the Bulk.execute() method. You can specify the write concern for the list in the Bulk.execute() method.
Once executed, you cannot re-execute the list without reinitializing.

For example,

var bulk = db.items.initializeUnorderedBulkOp();
bulk.insert( { _id: 1, item: "abc123", status: "A", soldQty: 5000 } );
bulk.insert( { _id: 2, item: "abc456", status: "A", soldQty: 150 } );
bulk.insert( { _id: 3, item: "abc789", status: "P", soldQty: 0 } );
bulk.execute( { w: "majority", wtimeout: 5000 } );


Common Commands


  1. user commands
    1. getLastError()
    2. isMaster()
    3. aggregation operations
    4. map-reduce
    5. count
    6. findAndModify: for complex updates. Nice way to do atomic, like, dequeuing of things from a queue
  2. dba commands
    1. drop collection
    2. create collection
    3. compact collection
    4. serverStatus
    5. replSetGetStatus()
    6. addShard



Which of the following are user commands (as opposed to admin commands)? Check all that apply.


db.runCommand()

db.runCommand(  {  << command name >> : <value>, param1: value1, param2 : value2} )

to get confirmations when the last operation was completed:

db.runCommand(  { "getLastError" : 1,
                                 "w"  : 2
                                 "wtimeout" : 3 }
                           )
When we execute an operation through the shell or another interface, we can use which of the following?
db.isMaster()

db.runCommand( "isMaster" )
db.isMaster()
db.runCommand( { "isMaster" : 1 } )  --> tell us if the server we are talking to is primary or not

What does the db.isMaster() command do?
db.serverStatus()

db.serverStatus() --> gives a lot of the statistics on what is happening on the server and a alot of those are used by the  MMS  (Mongo Monitoring Service)

When running the command db.serverStatus() in the shell, what does the “ok” field represent?
db.currentOp() & db.killOp()

db.currentOp() --> shows information of the current operation
db.killOp() --> to stop a long-running operation

If you’re looking for problems with database performance, what is a good place to look, when you run db.currentOp()?
ensureIndex(), getIndexes() & dropIndex()

db.collection.ensureIndex( { "name" : 1} ) --> to create a index
db.collection.getIndexes() --> get the indexes of a collection
db.collection.dropIndex(index) --> drop th indexes of a collection

db.products.find().explain() --> give us information about how the find was done

What will happen if an index is created on a field that does not exist in any of the documents in the collection?

collection.stats() & collection.drop()

db.collection.stats() --> give us statistical information about the collection
db.collection.drop() --> remove collection even though from namespaces

True or false: db.collection.remove({}), which removes all messages in a collection, is the same as db.collection.drop(), which drops the collection.
Review of Commands

Commands:

  1. Server
    1. isMaster
    2. serverStatus
    3. logout
    4. getLastError
  2. db
    1. dropDatabase
    2. repairDatabase
    3. clone
    4. copydb
    5. dbStats
  3. collection
    1. DBA
      1. create --> is implicit
      2. drop
      3. collstats
      4. rename collection
    2. user
      1. count
      2. aggregate
      3. MapReduce
      4. findAndModify
      5. geo*
  4. index
    1. ensureIndex
    2. getIndex

Which of these statements is true?

martes, 13 de enero de 2015

MongoDB course for developers. unit 5/8. Aggregation framework. Homeworks

Homework 5.1


Finding the most frequent author of comments on your blog
In this assignment you will use the aggregation framework to find the most frequent author of comments on your blog. We will be using a data set similar to ones we've used before. 

Start by downloading the handout zip file for this problem. Then import into your blog database as follows:

mongoimport -d blog -c posts --drop posts.json
Now use the aggregation framework to calculate the author with the greatest number of comments.

To help you verify your work before submitting, the author with the fewest comments is Mariela Sherer and she commented 387 times. 

db.posts.aggregate([
 { "$project" : { "author" : "$comments.author"}}
,{ "$unwind"  : "$author"}
,{ "$group"   : { "_id"            : "$author",
   "numPosts" : { "$sum" : 1} }}
,{ "$sort" : { "numPosts" :-1}}
,{"$limit" : 1}
])

Please choose your answer below for the most prolific comment author:

Homework 5.2

Crunching the Zipcode dataset
Please calculate the average population of cities in California (abbreviation CA) and New York (NY) (taken together) with populations over 25,000. 

For this problem, assume that a city name that appears in more than one state represents two separate cities. 

Please round the answer to a whole number. 
Hint: The answer for CT and NJ (using this data set) is 38177. 

Please note:

  • Different states might have the same city name.
  • A city might have multiple zip codes.


For purposes of keeping the Hands On shell quick, we have used a subset of the data you previously used in zips.json, not the full set. This is why there are only 200 documents (and 200 zip codes), and all of them are in New York, Connecticut, New Jersey, and California. 

If you prefer, you may download the handout and perform your analysis on your machine with

> mongoimport -d test -c zips --drop small_zips.json


db.zips.aggregate([
 { "$match" : { "$or" : [ { "state" : "CA" },{ "state" :"NY" } ] }}

,{ "$group" : { "_id" : { "state" : "$state", "city" : "$city"}, 
                  "pop"  : { "$sum" : "$pop"}} }

,{ "$match" : { "pop" : { "$gt" : 25000 }}}

,{ "$group" : { "_id" : null,
"avg" : { "$avg" : "$pop"}} }
])

Once you've generated your aggregation query and found your answer, select it from the choices below. 

Homework 5.3

Who's the easiest grader on campus?
A set of grades are loaded into the grades collection. 

The documents look like this:

{
 "_id" : ObjectId("50b59cd75bed76f46522c392"),
 "student_id" : 10,
 "class_id" : 5,
 "scores" : [
  {
   "type" : "exam",
   "score" : 69.17634380939022
  },
  {
   "type" : "quiz",
   "score" : 61.20182926719762
  },
  {
   "type" : "homework",
   "score" : 73.3293624199466
  },
  {
   "type" : "homework",
   "score" : 15.206314042622903
  },
  {
   "type" : "homework",
   "score" : 36.75297723087603
  },
  {
   "type" : "homework",
   "score" : 64.42913107330241
  }
 ]
}
There are documents for each student (student_id) across a variety of classes (class_id). Note that not all students in the same class have the same exact number of assessments. Some students have three homework assignments, etc. 

Your task is to calculate the class with the best average student performance. This involves calculating an average for each student in each class of all non-quiz assessments and then averaging those numbers to get a class average. To be clear, each student's average includes only exams and homework grades. Don't include their quiz scores in the calculation. 

What is the class_id which has the highest average student perfomance? 

Hint/Strategy: You need to group twice to solve this problem. You must figure out the GPA that each student has achieved in a class and then average those numbers to get a class average. After that, you just need to sort. The class with the lowest average is the class with class_id=2. Those students achieved a class average of 37.6 

If you prefer, you may download the handout and perform your analysis on your machine with

> mongoimport -d test -c grades --drop grades.json

db.grades.aggregate([

 { "$unwind" : "$scores" }

,{ "$match"  : { "$or"    : [ { "scores.type" : "exam" },{ "scores.type" : "homework" }]} }

,{ "$group"  : { "_id"    : { "class" : "$class_id", "student" : "$student_id" },
       "stdAvg" : {"$avg" : "$scores.score" } } }

,{ "$group"  : { "_id"    : "$_id.class",
       "avg1"   : { "$avg" : "$stdAvg"}  } }

,{ "$project" : { "_id"   : false, "class": "$_id", "avg" : "$avg1" } }

,{ "$sort"   : { "avg"    : -1} }

,{ "$limit"  : 1}
]) 

Below, choose the class_id with the highest average student average.





Homework 5.4

Removing Rural Residents
In this problem you will calculate the number of people who live in a zip code in the US where the city starts with a digit. We will take that to mean they don't really live in a city. Once again, you will be using the zip code collection, which you will find in the 'handouts' link in this page. Import it into your mongod using the following command from the command line:
> mongoimport -d test -c zips --drop zips.json

If you imported it correctly, you can go to the test database in the mongo shell and conform that
> db.zips.count()

yields 29,467 documents. 

The project operator can extract the first digit from any field. For example, to extract the first digit from the city field, you could write this query:
db.zips.aggregate([
    {$project: 
     {
 first_char: {$substr : ["$city",0,1]},
     }  
   }
])
Using the aggregation framework, calculate the sum total of people who are living in a zip code where the city starts with a digit. Choose the answer below. 

Note that you will need to probably change your projection to send more info through than just that first character. Also, you will need a filtering step to get rid of all documents where the city does not start with a digital (0-9).

db.zips.aggregate([
 { "$project" : { "fc"   : { "$substr" : ["$city",0,1]},           
"pop"  : "$pop" }}
,{ "$match"   : { "fc" : /^[0123456789]/   }}
,{ "$group"   : { "_id" :  null,
   "pop1" : { "$sum" : "$pop"}}}
,{ "$project" : { "_id" : false, "pop" : "$pop1"}}
])


miércoles, 7 de enero de 2015

MongoDB for DBA's 1/7. Introduction. Homeworks

Homework 1.1

Download and install MongoDB from www.mongodb.org. Then run the database as a single server instance on your PC (that is, run the mongod binary). Then, run the administrative shell.





From the shell prompt type
 db.isMaster().maxBsonObjectSize
at the ">" prompt.

What do you get as a result?
 16777216

Homework 1.2

Download the file products.json from education.mongodb.com. Take a look at its content.
Now, import its contents into MongoDB, into a database called "pcat" and a collection called "products". Use the mongoimport utility to do this.

mongoimport --host localhost --port 27017 --db pcat --collection products --file "products.json" --drop
When done, run this query in the mongo shell:
db.products.find({type:"case"}).count()
What's the result?
3

Homework 1.3

At this point you should have pcat.products loaded from the previous step. You can confirm this by running in the shell:
> db.products.find()
> // or:
> db.products.count()
> // should print out "11"
Now, what query would you run to get all the products where brand equals the string “ACME”?
db.products.find({"brand": "ACME"})

Homework 1.4

How would you print out, in the shell, the name of all the products without extraneous characters or braces, sorted alphabetically, ascending? (Check all that would apply.)


MongoDB for DBA's 1/7. Introduction

Chapter 1. Introduction to MongoDB, key concepts


Concepts







What were the big differences in hardware over the last few decades that MongoDB attempted to address?



Scaling







Q: When scaling out horizontally (adding more servers to contain your data), what are problems that arise as you go from, say, 1 commodity server to a few dozen?

SQL and Complex Transactions







What causes significant problems for SQL when you attempt to scale horizontally (to multiple servers)?

Documents Overview







What are some advantages of representing our data using a JSON-like format?



JSON Types







How many data types are there in JSON?

  1. String
  2. Integer
  3. Boolean
  4. Null
  5. Array
  6. Object or documents (subdocuments)

JSON Syntax







What is the corresponding JSON for the following XML document?
<person>
  <name>John</name>
  <age>25</age>
  <address>
    <city>New York</city>
    <postalCode>10021</postalCode>
  </address>
  <phones>
    <phone type="home">212-555-1234</phone>
    <phone type="mobile">646-555-1234</phone>
  </phones>
</person>


{"name":"John",
  "age":25,
  "address":{"city":"New York","postalCode":"10021"},
  "phones":[
                {"phone":"212-555-1234","type":"home"},
                {"phone":"646-555-1234","type":"mobile"}
                ]
}

JSON Syntax 2







For the following XML, Is the corresponding JSON example legal json?
<things>
  <hat>one</hat>
  <coat>z</coat>
  <hat>two</hat>
</things>
{
  "hat" : "one",
  "coat" : "z",
  "hat" : "two"
}


Binary JSON







Why do we represent our data as BSON rather than JSON in the system?


BSON and applications







For a typical client (a python client, for example) that is receiving the results of a query in BSON, would we convert from BSON to JSON to the client's native data structures (for example, nested dictionaries and lists in Python), or would we convert from BSON straight to those native data structures?


Dynamic Schema







True or False: MongoDB is schemaless because a schema isn't very important in MongoDB


What is the MongoDB shell?







Q: By default, which database does the mongo shell connect to?


Cursors Introduction







In order to query a collection in the mongo shell, we can type which of the following?


Query Language: Basic concepts

 
Mongo querys are represented as BSON (JSON)



You have a collection where every document has the same fields, and you want to look at the value of the “_id”, "name", and “email” fields in order to see their format. Furthermore, you want to eliminate all other fields from the query results. What query might you write?







Query Language: Projection

 
You want to query the “people” collection, you want the results of the query to include only documents where age is 50, and you want to look at all fields except “email”. What query should you write?



Query Language: Advantages of a Dynamic Schema






If you want to add a new key: value pair to the documents in the “shapes” collection, what methods could you use?


Shell: Queries






We have sample documents in our products collection such as:
{
  name: "AC1 Case Green",
  color: "green",
  price: 12.00,
  for: "ac1",
  type: ["accessory", "case"],
  available: true
}
How would we query in the shell for all products that are cases for an ac9 phone? That is, where type contains the value "case" and for equals "ac9"?


db.products.find({"for":"ac9","type":"case"})

Sorting






If you want to run a query on the collection, “books,” and sort ASCIIbetically by title on the query, which of the following will work?


Query Language: Cursors






Recall the documents in the scores collection:
{
 "_id" : ObjectId("50844162cb4cf4564b4694f8"),
 "student" : 0,
 "type" : "exam",
 "score" : 75
}
Write a query that retrieves documents of type "exam", sorted by score in descending order, skipping the first 50 and showing only the next 20.

db.scores.find({"type":"exam"}).sort({"score":-1}).skip(50).limit(20)