How to use find() nested documents for two levels or more?

Question

Here is my sample mongodb database

database image for one object

The above is a database with an array of articles. I fetched only one object for simplicity purposes.

database image for multiple objects ( max 20 as it's the size limit )

I have about 18k such entries. I have to extract the description and title tags present inside the (articles and 0) subsections. The find() method is the question here.. i have tried this :

for i in db.ncollec.find({'status':"ok"}, { 'articles.0.title' : 1 , 'articles.0.description' : 1}):
    for j in i:
        save.write(j)

After executing the code, the file save has this :

_id
articles
_id
articles

and it goes on and on..

Any help on how to print what i stated above?

My entire code for reference :

    import json
    import newsapi
    from newsapi import NewsApiClient
    import pymongo
    from pymongo import MongoClient

    client = MongoClient()
    db = client.dbasenews
    ncollec = db.ncollec


    newsapi = NewsApiClient(api_key='**********')
    source = open('TextsExtractedTemp.txt', 'r')
    destination = open('NewsExtracteddict.txt', "w")
    for word in source:
        if word == '\n':
            continue
        all_articles = newsapi.get_everything(q=word, language='en', page_size=1)
        print(all_articles)
        json.dump(all_articles, destination)
        destination.write("\n")
        try:
            ncollec.insert(all_articles)
        except:
            pass

Answer 1

Okay, so I checked a little to update my rusty memory of pymongo, and here is what I found.

The correct query should be :

db.ncollec.find({ 'status':"ok", 
                  'articles.title' : { '$exists' : 'True' },
                  'articles.description' : { '$exists' : 'True' } })

Now, if you do this :

query = { 'status' : "ok",
          'articles.title' : { '$exists' : 'True' },
          'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
    print item

And that it doesn't show anything, the query is correct, but you don't have the right database, or the right tree, or whatever.

But I assure you, that with the database you showed me, that if you do...

query = { 'status' : "ok",
          'articles.title' : { '$exists' : 'True' },
          'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
    save.write(item[0]['title'])
    save.write(item[0]['description'])

It'll do what you wished to do in the first place.

Now, the key item[0] might not be good, but for this, I can't really be of any help since it is was you are showing on the screen. :)

---

Okay, now. I have found something for you that is a bit more complicated, but is cool :) But I'm not sure if it'll work for you. I suspect you're giving us a wrong tree, since when you do .find( {'status' : 'ok'} ), it doesn't return anything, and it should return all the documents with a 'status' : 'ok', and since you have lots...

Anyways, here is the query, that you should use with .aggregate() method, instead of .find() :

elem = { '$match' : { 'status' : 'ok', 'articles.title' : { '$exists' : 'True'}, 'articles.description' : { '$exists' : 'True'}} }
[ elem, { '$unwind' : '$articles' }, elem ]

If you want an explanation as to how this works, I invite you to read this page.

This query will return ONLY the elements in your array that have a title, and a description, with a status OK. If an element doesn't have a title, or a description, it will be ignored.