Python. Listas, diccionarios, bucles y funciones

#############################################################
# listas
#############################################################

a = ['juan','josep','miquel']
# lista inicializada vacía
b = []
c = ["naranja", "pera", [1,2,3,4]]
d = [1,2,3]
e = [1, ['apple','pear'],3]
print(e[1])

#############################################################
# list slices
#############################################################
# a[ini:fin]  fin exclusivo (no lo obtiene)

a = [0,1,2,3,4]
# a[2:4] --> 2,3
a[2:] --> 2,3,4
a[:4] --> 0,1,2,3
a[:] -_> 0,1,2,3,4

#############################################################
# lista, inclusión
#############################################################

a['uno', 'dos', 'tres']
'uno' in a --> True
if 'uno' in a:
 print("es un uno")

#############################################################
# diccionarios
#############################################################

a = { 'nom' : 'joan', ocios : [ 'leer','natacion','bici']}
a['nom'] 
# --> 'joan'
a['ocios'][1]
# --> 'natacion'
a['ocios'].append('pasear')
# --> el resultado
# No conserva el orden de las claves de los diccionarios (documentos). 
# Después de ejecutar alguna operación puede que el diccionario quede ordenado diferente
a = { 'nom' : 'joan', ocios : [ 'leer','natacion','bici','pasear']}

# informa de todas las claves del diccionario
a.keys() --> ['nom', 'ocios']
# borra un elemento del diccionario
del(a['nom'])
# evalua si el string está en la lista de claves
'ocios' in a 

#############################################################
# Bucle FOR en lista
#############################################################
# muy importante la indentación, con ella identifica si está dentro del bucle o no
# para mi no es nada visual

fruit = ['apple', 'pear', 'banana']
new_fruit = []

for item in fruit:
 print item
 
 new_fruit.append(item)

print new_fruit

# resultado -->
# apple
# pear
# banana
# ['apple', 'pear', 'banana']


fruit = ['apple', 'pear', 'banana']
new_fruit = []

for item in fruit:
 print item
 
new_fruit.append(item)

print new_fruit

# resultado -->
# apple
# pear
# banana
# ['banana']

#############################################################
# bucle FOR en diccionario
#############################################################

print("bucle FOR en dicionario")
alumno = {'nom' : 'joan', 'apellido1' :'perez', 'apellido2' :'lopez'}

for key in alumno:
 print('la clave es: ' + key + ' y el valor es: ' + alumno[key])

# la salida es la siguiente y no está en el mismo orden que la defición
# bucle FOR en dicionario
# la clave es: apellido2 y el valor es: lopez
# la clave es: nom y el valor es: joan
# la clave es: apellido1 y el valor es: perez

#############################################################
# bucle WHILE en lista
#############################################################

print("\n\n###############  bucle WHILE en lista")
fruit = ['apple', 'pear', 'banana']
new_fruit = []
i = 0
while (i < len(fruit)):
 new_fruit.append(fruit[i])
 print( fruit[i] )
 i = i + 1

print(new_fruit)

# resultado -->
# apple
# pear
# banana
# ['apple', 'pear', 'banana']

#############################################################
# funciones
#############################################################

def analizaLista( lista ):
 
 counts = {}
 for item in lista:
  if item in counts:
   counts[item] = counts[item] + 1
  else:
   counts[item] = 1

 return counts

counts = analizaLista( frutas )
print(counts)

# resultado
# {'manzana': 1, 'pera': 2, 'platano': 1, 'naranja': 1}

#############################################################
# manejador de excepciones
#############################################################

print (5/0)

print("pero la vida continua")

MongoDB for developers 4/8. Performance. Homeworks

Homework 4.1

Suppose you have a collection with the following indexes:

> db.products.getIndexes()
[
 {
  "v" : 1,
  "key" : {
   "_id" : 1
  },
  "ns" : "store.products",
  "name" : "_id_"
 },
 {
  "v" : 1,
  "key" : {
   "sku" : 1
  },
                "unique" : true,
  "ns" : "store.products",
  "name" : "sku_1"
 },
 {
  "v" : 1,
  "key" : {
   "price" : -1
  },
  "ns" : "store.products",
  "name" : "price_-1"
 },
 {
  "v" : 1,
  "key" : {
   "description" : 1
  },
  "ns" : "store.products",
  "name" : "description_1"
 },
 {
  "v" : 1,
  "key" : {
   "category" : 1,
   "brand" : 1
  },
  "ns" : "store.products",
  "name" : "category_1_brand_1"
 },
 {
  "v" : 1,
  "key" : {
   "reviews.author" : 1
  },
  "ns" : "store.products",
  "name" : "reviews.author_1"
 }

 Which of the following queries can utilize an index. Check all that apply.

db.products.find({'brand':"GE"})

db.products.find({'brand':"GE"}).sort({price:1})

db.products.find({$and:[{price:{$gt:30}},{price:{$lt:50}}]}).sort({brand:1})

db.products.find({brand:'GE'}).sort({category:1, brand:-1}).explain()

Homework 4.2

Suppose you have a collection called tweets whose documents contain information about the created_at time of the tweet and the user's followers_count at the time they issued the tweet. What can you infer from the following explain output?

db.tweets.find({"user.followers_count":{$gt:1000}}).sort({"created_at" : 1 }).limit(10).skip(5000).explain()
{
        "cursor" : "BtreeCursor created_at_-1 reverse",
        "isMultiKey" : false,
        "n" : 10,
        "nscannedObjects" : 46462,
        "nscanned" : 46462,
        "nscannedObjectsAllPlans" : 49763,
        "nscannedAllPlans" : 49763,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 205,
        "indexBounds" : {
                "created_at" : [
                        [
                                {
                                        "$minElement" : 1
                                },
                                {
                                        "$maxElement" : 1
                                }
                        ]
                ]
        },
        "server" : "localhost.localdomain:27017"
}

This query performs a collection scan.

The query uses an index to determine the order in which to return result documents.

The query uses an index to determine which documents match.

The query returns 46462 documents.

The query visits 46462 documents.

The query is a "covered index query".

Homework 4.3

Making the Blog fastPlease download hw4-3.zip from the Download Handout link to get started. This assignment requires Mongo 2.2 or above.
In this homework assignment you will be adding some indexes to the post collection to make the blog fast.
We have provided the full code for the blog application and you don't need to make any changes, or even run the blog. But you can, for fun.
We are also providing a patriotic (if you are an American) data set for the blog. There are 1000 entries with lots of comments and tags. You must load this dataset to complete the problem.

# from the mongo shell
use blog
db.posts.drop()
# from the a mac or PC terminal window
mongoimport -d blog -c posts < posts.json

or

mongoimport --host localhost --port 27017 --db blog --collection posts --file "posts.json" --drop --stopOnError

The blog has been enhanced so that it can also display the top 10 most recent posts by tag. There are hyperlinks from the post tags to the page that displays the 10 most recent blog entries for that tag. (run the blog and it will be obvious)
Your assignment is to make the following blog pages fast:

• The blog home page
• The page that displays blog posts by tag (http://localhost:8082/tag/whatever)
• The page that displays a blog entry by permalink (http://localhost:8082/post/permalink)

By fast, we mean that indexes should be in place to satisfy these queries such that we only need to scan the number of documents we are going to return. To figure out what queries you need to optimize, you can read the blog.py code and see what it does to display those pages. Isolate those queries and use explain to explore.

****************************

    # returns an array of num_posts posts, reverse ordered
    def get_posts(self, num_posts):

##################################################################
        self.posts.ensure_index([ ("date", pymongo.DESCENDING)])
##################################################################        

        cursor = self.posts.find().sort('date', direction=-1).limit(num_posts)
        l = []

        for post in cursor:
            post['date'] = post['date'].strftime("%A, %B %d %Y at %I:%M%p") # fix up date
            if 'tags' not in post:
                post['tags'] = [] # fill it in if its not there already
            if 'comments' not in post:
                post['comments'] = []

            l.append({'title':post['title'], 'body':post['body'], 'post_date':post['date'],
                      'permalink':post['permalink'],
                      'tags':post['tags'],
                      'author':post['author'],
                      'comments':post['comments']})

        return l

    # returns an array of num_posts posts, reverse ordered, filtered by tag
    def get_posts_by_tag(self, tag, num_posts):

##################################################################
        self.posts.ensure_index([ ("tags", pymongo.ASCENDING),("date", pymongo.DESCENDING)])
##################################################################        

        cursor = self.posts.find({'tags':tag}).sort('date', direction=-1).limit(num_posts)
        l = []

        for post in cursor:
            post['date'] = post['date'].strftime("%A, %B %d %Y at %I:%M%p")     # fix up date
            if 'tags' not in post:
                post['tags'] = []           # fill it in if its not there already
            if 'comments' not in post:
                post['comments'] = []

            l.append({'title': post['title'], 'body': post['body'], 'post_date': post['date'],
                      'permalink': post['permalink'],
                      'tags': post['tags'],
                      'author': post['author'],
                      'comments': post['comments']})

        return l

    # find a post corresponding to a particular permalink
    def get_post_by_permalink(self, permalink):

##################################################################
        self.posts.ensure_index([ ("permalink", pymongo.ASCENDING)])
##################################################################        

        post = self.posts.find_one({'permalink': permalink})

        if post is not None:
            # fix up likes values. set to zero if data is not present
            for comment in post['comments']:
                if 'num_likes' not in comment:
                    comment['num_likes'] = 0

            # fix up date
            post['date'] = post['date'].strftime("%A, %B %d %Y at %I:%M%p")

        return post

Once you have added the indexes to make those pages fast run the following.

python validate.py

(note that for folks who are using MongoLabs or MongoHQ there are some command line options to validate.py to make it possible to use those services) Now enter the validation code below.

Homework 4.4

In this problem you will analyze a profile log taken from a mongoDB instance. To start, please download sysprofile.json from Download Handout link and import it with the following command:

mongoimport -d m101 -c profile < sysprofile.json

or

mongoimport --host localhost --port 27017 --db m101 --collection profile --file "sysprofile.json" --drop --stopOnError

Now query the profile data, looking for all queries to the students collection in the database school2, sorted in order of decreasing latency.

db.profile.find({"ns" : /school2.students/}).sort({"millis":-1}).limit(1).pretty()

What is the latency of the longest running operation to the collection, in milliseconds?

19776550 10000000 5580001 15820 2350

MongoDB for developers 4/8. Performance

Indexes

Indexes are the most important factor in mongodb performance. By default the data is locked for sequently access.

• Use indexing in order to find sorted data is faster than not indexing
• The order of the keys in the index is important to find data fast
• Because indexes take space on disk and need to be updated every write, it is important not to index for all keys of the document- It is much more efficiently to have indexes for all the most common queries
• We can use the keys of index in the order of are defined. index (a,b,c) --> index for a, index for a,b but not index for c or index for c.

The optimization that have the greatest impact on the performance of a database is adding appropiate indexes on large collections so that only a small percentatge of queries need to scan the collection

Creating Indexes

• db.collection.ensureIndex({ camp: order (1 or -1)})
• db.system.indexes.find()     --> show all the indexes
• db.collection.getIndexes()   --> show the indexes of a collection
• db.collection.dropIndex({}) --> drop an index of a collection

Please provide the mongo shell command to add an index to a collection namedstudents, having the index key be class, student_name.

db.students.ensureIndex({class:1, student_name:1})

Multikey Indexes

mongodb supports:  an index on a key with value can be an array

mongodb does not support:  an index with combinations of a key with value of an array and other document's elements.

 

An index with more than one combination of arrays and values than are not arrays:
            ensureIndex({a:1, b:1})
            {a:1,b:1}
            {a:[1,2,3] , b:1}        --> support
            {a:[1,2,3] , b:[4,5,6]} --> not support
 

db.collection.find().explain() --> show ahow the find has been made 

 

Suppose we have a collection foo that has an index created as follows:

db.foo.ensureIndex({a:1, b:1})

Which of the following inserts are valid to this collection?

db.foo.insert({a:["apples","oranges"], b:"grapes"})

db.foo.insert({a:"grapes", b:"oranges"})

db.foo.insert({a:"grapes", b:[8,9,10]})

db.foo.insert({a:[1,2,3], b:[5,6,7]}) 

we can make an index of subparts of an array:

        b: [ {a:1,b:1, c: [1,2,3] }]

Two parallel arrays index are not allowed.

Index Creation option, Unique

Please provide the mongo shell command to add a unique index to the collectionstudents on the keys student_id, class_id.

db.students.ensureIndex({student_id:1, class_id:1}, {unique: true})

Index Creation, Removing Dups

db.collection.ensureIndex( {a:1},{unique: true, dropDups:true}}) --> when the index is created, if it finds a dupplicated document, remove all documents that have this dupplicated key, except one

If you choose the dropDups option when creating a unique index, what will the MongoDB do to documents that conflict with an existing index entry?

Delete them for ever and ever, Amen.

Index Creation, Sparse

To create unique indexes when the indexed key is not present in the document.
    1. {a:1,b:2,c:3}
    2. {a:10,b:5,c:10}
    3. {a:13,b:4}
    4. {a:7,b:23}

In this documents, a Spare index will create a index with the present keys discarding the documents that do not contain the same key. If we want to index for {c:1}, the documents 3 and 4 will not be added to the index

• db.collection.ensureIndex( {a:1},{unique: true, sparse:true}}) --> it will create a disperse index
• db.collection.find().sort().hint() --> hint() forces the query optimizer to use a specific index to fulfill the query

Suppose you had the following documents in a collection called people with the following docs:

> db.people.find()
{ "_id" : ObjectId("50a464fb0a9dfcc4f19d6271"), "name" : "Andrew", "title" : "Jester" }
{ "_id" : ObjectId("50a4650c0a9dfcc4f19d6272"), "name" : "Dwight", "title" : "CEO" }
{ "_id" : ObjectId("50a465280a9dfcc4f19d6273"), "name" : "John" }

And there is an index defined as follows:

db.people.ensureIndex({title:1}, {sparse:1})

If you perform the following query, what do you get back, and why?

db.people.find({title:null})

No documents, because the query uses the index and there are no documents with title:null in the index.

Index Creation, Background

foreground (default)    Background:
        faster                     slow
        block writes            dos not block writers
            (per DBlock)

Which things are true about creating an index in the background in MongoDB. Check all that apply.

A mongod instance can only build one background index at a time per database.

Although the database server will continue to take requests, a background index creation still blocks the mongo shell that you are using to create the index.

Creating an index in the background takes longer than creating it in the foreground

In Mongo 2.2 and above, indexes are created in the background by default.  

Using Explain

Inform how the query was done,which index was used to and how they were used.

Given the following output from explain, what is the best description of what happened during the query?

{
 "cursor" : "BasicCursor",
 "isMultiKey" : false,
 "n" : 100000,
 "nscannedObjects" : 10000000,
 "nscanned" : 10000000,
 "nscannedObjectsAllPlans" : 10000000,
 "nscannedAllPlans" : 10000000,
 "scanAndOrder" : false,
 "indexOnly" : false,
 "nYields" : 7,
 "nChunkSkips" : 0,
 "millis" : 5151,
 "indexBounds" : {
  
 },
 "server" : "Andrews-iMac.local:27017"
}

The query scanned 10,000,000 documents, returning 100,000 in 5.2 seconds.

When is an index used?

MongoDb extract estatistic information of the useful queries and choose the best indexation in background every 100 queries more or less.

Given collection foo with the following index:

db.foo.ensureIndex({a:1, b:1, c:1})

Which of the following queries will use the index?

db.foo.find({b:3, c:4})

db.foo.find({a:3})

db.foo.find({c:1}).sort({a:1, b:1}) 

db.foo.find({c:1}).sort({a:-1, b:1}) 

How large is your index?

Indexes have to be in memory in order to get good performance. The size of the index can be very big and will use a lot of memory. This is a consideration at time to planning what sort of indexes we want to create for the documents that we have.

• db.collection.stats()              --> statistic information
• db.collection.totalIndexSize() --> get information of size on disc of indexes

Is it more important that your index or your data fit into memory?

Index Data

Index Cardinality

• Regular index: 1 to 1
• Sparse index: <= documents
• Multikey index: with array of tags  > number of documents

Let's say you update a document with a key called tags and that update causes the document to need to get moved on disk. If the document has 100 tags in it, and if the tags array is indexed with a multikey index, how many index points need to be updated in the index to accomodate the move?

100

Indexing in pyMongo

db.collection.ensureIndex([ ('key1', pymongo.ASCENDING), ('key2', pymongo.DESCENDING)])   

Hinting an Index

• db.people.find().sort({'title':1}).hint({'title:1}
• db.people.find().sort({'title':1}).hint({ $natural:1 }) --> specify the index which is the best for mongodb 

hint() specify wich index will be used. Using an index with a key that do not exist in the documents, the query cannot be executed because there is not any pointer in the index to any document.
 

Given the following data in a collection:

> db.people.find()
{ "_id" : ObjectId("50a464fb0a9dfcc4f19d6271"), "name" : "Andrew", "title" : "Jester" }
{ "_id" : ObjectId("50a4650c0a9dfcc4f19d6272"), "name" : "Dwight", "title" : "CEO" }
{ "_id" : ObjectId("50a465280a9dfcc4f19d6273"), "name" : "John" }

and the following indexex:

> db.people.getIndexes()
[
 {
  "v" : 1,
  "key" : {
   "_id" : 1
  },
  "ns" : "test.people",
  "name" : "_id_"
 },
 {
  "v" : 1,
  "key" : {
   "title" : 1
  },
  "ns" : "test.people",
  "name" : "title_1",
  "sparse" : 1
 }
]

Which query below will return the most documents.

db.people.find().sort({'title':1}).hint({$natural:1})db.people.find().sort({'title':1})db.people.find({name:{$ne:"Kevin"}}).sort({'title':1})db.people.find({'title':{$ne:null}}).hint({'title':1})

hint natural to use BasicCursor returns all docs.

Efficiency of index use

There are elements that $gt, $lt, $eq, $ne, $exist  than can make the query slow because have to examine all the documents.

Is better to use regular expressions /abcd/ -> look for a,b,c,d, /^abcd/ do not look for a,b,c,d
 

Keep in mind when you think aboinut indexing you have to consider how the index was used: only for the sort o if it was used inefficiently and caused that de database examined millions of records, etc

  

Geospatial Indexes

They are indexes based in locations using 2D coordinates. : {'location': [x,y] }
 

ensureIndex({ "location": '2d', type: 1})
find({location: { "$near" : [x,y] }} ) --> retorn locatiosn in increase distances

 

Suppose you have a 2D geospatial index defined on the key location in the collection places. Write a query that will find the closest three places (the closest three documents) to the location 74, 140.

 

db.places.find({location: {$near: [74,140]}}).limit(3)

Geospatial Spherical

• lng -> vertical 
• lat  -> horizontal (-90 to 90)

specification GeoJSON -> ( )
     { "location" : { Type : "Point", "coordinates : [-122,40] "} }

 

ensureIndex( { location : '2dsphere'})
 

find( { "location" :
            { "$near" :
                { "$geometry" :
                    { "type"         : "Point" ,
                      "coordinates"  : [-10,10] },
                      "$maxdistante" : 2000 <-- in meters
                     }
                }
            })

What is the query that will query a collection named "stores" to return the stores that are within 1,000,000 meters of the location latitude=39, longitude=-130? Type the query in the box below. Assume the stores collection has a 2dsphere index on "loc" and please use the "$near" operator. Each store record looks like this: 

 

{ "_id" : { "$oid" : "535471aaf28b4d8ee1e1c86f" },

  "store_id" : 8, 

  "loc" : { "type" : "Point", "coordinates" : [ -37.47891236119904, 4.488667018711567 ] } }

 

db.stores.find( { loc : { "$near" : { "$geometry" : { "type" : "Point", "coordinates : [ -130, 39]},"$maxdistance" : 1000000}}})

Full Text searches in mongoDb

There is a type of index that allow to look for text in the data.

 

ensureIndex( { 'words': 'text'})

 

db.collection.find( { "$text" : {"$search":'texto'}) --> look for dog in the documents no case-sensitive.
 

db.collection.find( { "$text" : {"$search":'word1 word2 word3 '}}, { "score" : {"$meta" : 'textScore'}}).sort( { "score": { "$meta" : 'textScore'}}) --> look for documents that contains all of the tree wordsYou create a text index on the "title" field of the movies collection, and then perform the following text search:

> db.movies.find( { $text : { $search : "Big Lebowski" } } )

Which of the following documents will be returned, assuming they are in the movies collection? Check all that apply.

{ "title" : "The Big Lebowski" , star: "Jeff Bridges" }

{ "title" : "Big" , star : "Tom Hanks" }

{ "title" : "Big Fish" , star: "Ewan McGregor" }

Logging and profiling: log slow queries

Mongodb have a profiler to detect via log informing how mongod is accessing to database:  system.profile
 

We have tree levels to log information of tue queries to know how our application is working:

• level 0: default and it is log off
• level 1: only log slow queries (register slow queries)
• level 2: record all logs of the queries   (register my queries) --> is for debugging

mongod --db dbpath --profile 1 --slows 2 (2 mseconds)

• db.system.profile.find()
• db.getProfilingLevel()
• db.getProfilingStatus()
• db.setProfilingLevel(1,4)  level 1 , 4 mseconds
• db.setProfilingLevel(0) --> off

Write the query to look in the system profile collection for all queries that took longer than one second, ordered by timestamp descending.

db.system.profile.find({millis: {$gt: 1000}}).sort({ts: -1})

MongoStat

system information of mongodb database.

    column idx miss% --> % de perdida de memoria por los índices

  

MongoTop

 give a high level view of how mongdb is spending the time.

 

Resume

    1. indexes are critical to performance
    2. explain()
    3. hint()
    4. profiling

 

Sharding

It is a techique to divide a collections in multiples servers
 
Application --> mongos --> mongod 1
                                       --> mongod 2
                                       --> mongod 3
 
It is necessaryt to include a sharding key to look for the server in which document is

MongoDB for developers 3/8. Schema design. Homeworks

Homework 3.1

Download the students.json file from the Download Handout link and import it into your local Mongo instance with this command:

$ mongoimport --host localhost --port 27017 --db school --collection students 

--file "students.json" --drop --stopOnError

or

$ mongoimport -d school -c students < students.json

This dataset holds the same type of data as last week's grade collection, but it's modeled differently. You might want to start by inspecting it in the Mongo shell.
Write a program in the language of your choice that will remove the lowest homework score for each student. Since there is a single document for each student containing an array of scores, you will need to update the scores array and remove the homework.
Remember, just remove a homework score. Don't remove a quiz or an exam!
Hint/spoiler: With the new schema, this problem is a lot harder and that is sort of the point. One way is to find the lowest homework in code and then update the scores array with the low homework pruned. 

import pymongo
import sys


# Copyright 2014
# Author: JJFT

# connnecto to the db on standard port
connection = pymongo.Connection("mongodb://localhost", safe=True)

db = connection.school       # attach to db
collection = db.students         # specify the colllection

print db
print collection

query = {'scores.type':'homework'}

try:
    cursor = collection.find(query)
    for doc in cursor:
        scores = doc['scores']
        idd = doc['_id']
        print idd
        scoAnt = 0
        for scr in scores:
            if (scr['type'] == "homework"):
                if (scoAnt == 0): scoAnt = scr['score']
                print scr,',',scr['type'],',',scr['score']
                if (scr['score'] < scoAnt): scoAnt = scr['score']
        print scoAnt
        cond = {"$and" : [{'_id' : idd},{ 'scores.type' : "homework"}] }
        query = { '$pull': { 'scores': { "score": scoAnt } } }
        collection.update( cond,query,multi=False)
        print doc 

except:
    # print ("Error trying to read collection:" + sys.exc_info()[0])
        print ("Error trying to read collection:")

To confirm you are on the right track, here are some queries to run after you process the data with the correct answer shown:
Let us count the number of students we have:

> use school
> db.students.count() 
200

Let's see what Tamika Schildgen's record looks like:

> db.students.find( { _id : 137 } ).pretty( )
{
 "_id" : 137,
 "name" : "Tamika Schildgen",
 "scores" : [
  {
   "type" : "exam",
   "score" : 4.433956226109692
  },
  {
   "type" : "quiz",
   "score" : 65.50313785402548
  },
  {
   "type" : "homework",
   "score" : 89.5950384993947
  }
 ]
}

To verify that you have completed this task correctly, provide the identity (in the form of their _id) of the student with the highest average in the class with following query that uses the aggregation framework. The answer will appear in the _id field of the resulting document.

> db.students.aggregate(

     { '$unwind' : '$scores' }

   , { '$group' : { '_id' : '$_id' , 'average' : { $avg : '$scores.score' } } }

   , { '$sort' : { 'average' : -1 } } , { '$limit' : 1 } ) 

13 

Homework 3.2

Making your blog accept posts In this homework you will be enhancing the blog project to insert entries into the posts collection. After this, the blog will work. It will allow you to add blog posts with a title, body and tags and have it be added to the posts collection properly.
We have provided the code that creates users and allows you to login (the assignment from last week). To get started, please download hw3-2and3-3.zip from the Download Handout link and unpack. You will be using these file for this homework and the HW 3.3.
The areas where you need to add code are marked with XXX. You need only touch the BlogPostDAO.py file. There are three locations for you to add code for this problem. Scan that file for XXX to see where to work. 

    # inserts the blog entry and returns a permalink for the entry
    def insert_entry(self, title, post, tags_array, author):
        print "inserting blog entry", title, post

        # fix up the permalink to not include whitespace

        exp = re.compile('\W') # match anything not alphanumeric
        whitespace = re.compile('\s')
        temp_title = whitespace.sub("_",title)
        permalink = exp.sub('', temp_title)

        # Build a new post
        post = {"title": title,
                "author": author,
                "body": post,
                "permalink":permalink,
                "tags": tags_array,
                "comments": [],
                "date": datetime.datetime.utcnow()}

        # now insert the post
        try:
            # XXX HW 3.2 Work Here to insert the post
            self.posts.insert(post)
            print "Inserting the post"
        except:
            print "Error inserting post"
            print "Unexpected error:", sys.exc_info()[0]

        return permalink

    # returns an array of num_posts posts, reverse ordered
    def get_posts(self, num_posts):

        cursor = []         # Placeholder so blog compiles before you make your changes

        # XXX HW 3.2 Work here to get the posts
        cursor = self.posts.find().limit(num_posts)

        l = []

        for post in cursor:
            post['date'] = post['date'].strftime("%A, %B %d %Y at %I:%M%p") # fix up date
            if 'tags' not in post:
                post['tags'] = [] # fill it in if its not there already
            if 'comments' not in post:
                post['comments'] = []

            l.append({'title':post['title'], 'body':post['body'], 'post_date':post['date'],
                      'permalink':post['permalink'],
                      'tags':post['tags'],
                      'author':post['author'],
                      'comments':post['comments']})

        return l

    # find a post corresponding to a particular permalink
    def get_post_by_permalink(self, permalink):

        post = None
        # XXX Work here to retrieve the specified post
        post = self.posts.find_one({'permalink':permalink})

        if post is not None:
            # fix up date
            post['date'] = post['date'].strftime("%A, %B %d %Y at %I:%M%p")

        return post

As a reminder, to run your blog you type

python blog.py

To play with the blog you can navigate to the following URLs

http://localhost:8082/
http://localhost:8082/signup
http://localhost:8082/login
http://localhost:8082/newpost

You will be proving that it works by running our validation script as follows:

python validate.py

You need to run this in a separate terminal window while your blog is running and while the database is running. It makes connections to both to determine if your program works properly. Validate connects to localhost:8082 and expects that mongod is running on localhost on port 27017.
As before, validate will take some optional arguments if you want to run mongod on a different host or a use an external webserver.
This project requires Python 2.7. The code is not 3.0 compliant.
Ok, once you get the blog posts working, validate.py will print out a validation code for HW 3.2.
Please enter it below, exactly as shown with no spaces.

Homework 3.3

Making your blog accept comments In this homework you will add code to your blog so that it accepts comments. You will be using the same code as you downloaded for HW 3.2.
Once again, the area where you need to work is marked with an XXX in the blogPostDAO.py file. There are one location. You don't need to figure out how to retrieve comments for this homework because the code you did in 3.2 already pulls the entire blog post (unless you specifically projected to eliminate the comments) and we gave you the code that pulls them out of the JSON document.
This assignment has fairly little code, but it's a little more subtle than the previous assignment because you are going to be manipulating an array within the Mongo document. For the sake of clarity, here is a document out of the posts collection from a working project.

 {
 "_id" : ObjectId("509df76fbcf1bf5b27b4a23e"),
 "author" : "erlichson",
 "body" : "This is a blog entry",
 "comments" : [
  {
   "body" : "This is my comment",
   "author" : "Andrew Erlichson"
  },
  {
   "body" : "Give me liberty or give me death.",
   "author" : "Patrick Henry"
  }
 ],
 "date" : ISODate("2012-11-10T06:42:55.733Z"),
 "permalink" : "This_is_a_blog_post_title",
 "tags" : [
  "cycling",
  "running",
  "swimming"
 ],
 "title" : "This is a blog post title"
}

Note that you add comments in this blog from the blog post detail page, which appears at

http://localhost:8082/post/post_slug

where post_slug is the permalink. For the sake of eliminating doubt, the permalink for the example blog post above is http://localhost:8082/post/This_is_a_blog_post_title 

    # add a comment to a particular blog post
    def add_comment(self, permalink, name, email, body):

        comment = {'author': name, 'body': body}

        if (email != ""):
            comment['email'] = email

        try:
            # this is here so the code runs before you fix the next line
            last_error = {'n':-1}
            # XXX HW 3.3 Work here to add the comment to the designated post
            
            self.posts.update({'permalink':permalink},
                   {"$push" : { "comments" : comment} },multi=False)

            return last_error['n']          # return the number of documents updated

        except:
            print "Could not update the collection, error"
            print "Unexpected error:", sys.exc_info()[0]
            return 0

You will run validation.py to check your work, much like the last problem. Validation.py will run through and check the requirements of HW 3.2 and then will check to make sure it can add blog comments, as required by this problem, HW 3.3. It checks the web output as well as the database documents.

python validate.py

Once you have the validation code, please copy and paste in the box below, no spaces.

MongoDB for developers 3/8. Schema design

MongoDB Schema Design

What's the single most important factor in designing your application schema within MongoDB?

Matching the data access patterns of your application.

Mongo Design for Blog

Which data access pattern is not well supported by the blog schema?

Providing a table of contents by tag (need aggregation to group it)

Living without Constraints

''Embeber primero los datos que tengan sentido para la aplicacion ya que hacen mucho mas facil mantener los datos intactos y consistentes."

 

What does Living Without Constraints refer to?

Living every day like it's your last

Saying whatever you want when you want it

Keeping your data consistent even though MongoDB lacks foreign key constraints

Wearing no belt

Living without Transactions

In the relational world, transactions offer atomicity, consistency, isolation and durability. In MongoDB have not transactions but we have atomic operations.

Atomic operation: we works in a single document, the works will completed before anyone else sees the document. They will see all the changes or none of them.
Doing atomic operations can get the same of transactions in retational databases and the reason is that in a relational database you can begin a transaction through some tables that needs to be join at a time and in mongoDB if you have all the data embedded, the transaction is also completed at a time.

In MongoDB we have three options to simulate transactions:

1. Reestructurate de code: in order to embedded data using only one transaction to get all the necessary data
2. Implement transactions in software: implementing critical sections to find and modify, semaphores, etc.
3. Tolerate consistency: often works in modern web apllications and others aplications that assume that in transmit data is just tolerate a little bit of inconsistency. Ex. People updating information in facebook can see information of their friends that has not yet updated.

Examples of operations that operate atomically within a single document:

Update

findAndModify

$addToSet (within an update)

$push within an update

Relationships. Denormalize

While we do not duplicate data, we will not be vulnerable to modify data.

One to One Relations: Always embedded

Considerations depending of access to data and hoe frequency access to a piece of the data (example of collections of employee and Resume (CV) )

• frequency access: to one document respect the other. If a document has a lot of information that do not need to update frequency you will choose separate documents in order to get best performance.
• if the size of the elements of one document are growing all the time or not, you can decide to separate the collections in order not to load a lot of information every time you can update some specific piece of information and of course if the info of the one document is larger of 16MB (multimedia info, events history, etc)
• atomicity of data: in mongoDb there is not transactions but only atomic operations in individual documents. If you do not accept any inconsistency and you want to update the separated documents all the same time, you can embedd the documents to update all in one.

 

The good reason to keep two documents that are related to each other one-to-one in separate collecions are:

Because you want to allow atomic update of both documents at once.

To reduce the working set size of your application.

To enforce foreign key constraints

Because the combined size of the documents would be larger than 16MB

Is perfectly secure to embed data because yo are not duplicating data

One To Many Relations

Only embedded if we have one to few that's much more easy to modelate in mongodb. it is recommended to represent a one to many relationship in multiple collections when the many is large
 

Embedding will work so good without data duplicated if embedding the many to the ones. If you want to go to the ones to the many, linking will prevent data duplicated.
 

If you need embed data due to performance of application dessign pattern thie will have sense if you have diplicity of data specially if your data changes continuesly or updates a lot

It is recommended to represent a one to many relationship in multiple collections:

Always

Whenever the many is large

Whenever the many is actually few

Never 

Many To Many Relations

If we have few to few we have to model in mongodb embedding documents.
 

To prevent problems with denormalization we have to make relationship linking with arrays of ObjectId in the documents.

Multikeys indexes

Arrays in the multikeys indexes allow find documents so fast

Benefits of embedding

The data in the disk is continuesly and the performance to read is high

Trees

mongodb can list ancestors and children. The best way is include an array in the document with the ancestors of this document sorted in order to find all the parent documents of the present document. Example:
 

Given the following typical document for a e-commerce category hierarchy collection called categories

 

{
  _id: 34,
  name : "Snorkeling",
  parent_id: 12,
  ancestors: [12, 35, 90]
}

 

Which query will find all descendants of the snorkeling category?

 

db.categories.find({ancestors:{'$in':[12,35,90]}})

db.categories.find({parent_id: 34})

db.categories.find({_id:{'$in':[12,35,90]}})

db.categories.find({ancestors:34}) 

When to denormalize

while we do not duplicate data, we will not be vulnerable to modify data

Handling Blobs

GridFS stores large files and blobs in two collections, one for metadata and one for the blob chunks.