2

I am working on an app that shows the user certain available services within their selected location, including the price of that service. I have Services stored in a collection, and the pricing of each of those services stored in another collection of their own (Prices). Some services have the same pricing, so there's often a one-to-many relationship.

The problem is that in my 'algorithm', which I use to determine the best available service, there are a lot of values that I determine by polling Mongo and getting the right value, which includes the price of the service.

I initially embedded the pricing structure of each service in the Service model, but that was a problem because I would have to scan through the Services collection regularly to update this pricing for changes/volatility. I ended up embedding a DB reference from the Service to the Price.

My Question

How can I best achieve what I am currently doing while reducing the number of calls to Mongo?

I'm currently hitting Mongo at about 50-400 reads (specifically for pricing) per user search, which slows things down a bit when combined with other queries, and is downright inefficient. I am considering a number of options:

  1. Creating a bucket (maybe an object in the global namespace in Node) where I cache all the pricing info by _id. Each time I need to find pricing for a Service, I go through that bucket, if the pricing has already been cached in the bucket, I do my calculations and return the price, else I fetch the document from Mongo and add it to the bucket for current and future use.

I am worried this might introduce problems such as memory leaks. Ideally the bucket should be available to all the users, not like a cookie that belongs only to one user (my understanding of cookies might be wrong here). I don't yet know if it's possible to expose an object globally for all users to have access to it.

  1. I have seen a number of those 'in memory implementations' of Mongo (can't find any at the moment), which emulate MongoDB but using LocalStorage as a data store. I am considering forking one of them and tweaking them for server-side functionality.

I don't know if this could work, and it seems to be likely to take me some time to do.

  1. Store the data as a JSON file (or a number of files, I can work out a way of splitting the files). I can then read the JSON files with Node by require()ing them. This is currently my preferred option, as I can just use the fs API to handle the periodic updating of the JSON files.

I am concerned about the size of the JSON files, mainly that if they're sufficiently large, they might slow Node down, but I'm considering this as I'll be saving 50 - 400 hits on the DB. I am also assuming that I won't have a single JSON file loaded n times (n being the number of users on the service at a point in time)

Considering the above 3 options, which has a good chance of working? Some of the options might be impossible, so I can scratch them off my list, there might be others which I haven't yet considered.

Lastly, my disclaimer: I know that this could have been implemented easily in SQL using JOIN statements, and no I am not using a hammer for something that needs a screwdriver. Although for some parts of my project SQL looks like a clear win, for a large part of the data that I store, Mongo is better. Plus reinventing the wheel in the name of experience helps.

Thanks

nevi_me
  • 2,432
  • 3
  • 20
  • 31
  • One of good considerations would be to make mongodb to store data in memory rather than on hdd. This is possible as mongodb is IO agnostic. Check out this article: http://edgystuff.tumblr.com/post/49304254688/how-to-use-mongodb-as-a-pure-in-memory-db-redis-style – moka Aug 30 '13 at 12:22
  • I've seen the article, I don't want to complicate things for myself, at least yet. I have implemented the third option, and will put it in production in a few weeks. I'll make up my mind after that on which is better. As (if) I scale I'll look into storing everything in memory, which I think Mongo already does. – nevi_me Aug 30 '13 at 18:29
  • A very good consideration would be to use a cache like redis. They both play very well [together](http://stackoverflow.com/questions/10696463/mongodb-with-redis). – Pradeep Mahdevu Oct 08 '13 at 20:04
  • @PradeepMahdevu will certainly look into it, hadn't thought of that. I'll try it out one of these weeks, then come back and update this question with my results – nevi_me Oct 09 '13 at 07:55

0 Answers0