I am working on an app that shows the user certain available services within their selected location, including the price of that service. I have Services
stored in a collection, and the pricing of each of those services stored in another collection of their own (Prices
). Some services have the same pricing, so there's often a one-to-many relationship.
The problem is that in my 'algorithm', which I use to determine the best available service, there are a lot of values that I determine by polling Mongo and getting the right value, which includes the price of the service.
I initially embedded the pricing structure of each service in the Service
model, but that was a problem because I would have to scan through the Services
collection regularly to update this pricing for changes/volatility. I ended up embedding a DB reference from the Service
to the Price
.
My Question
How can I best achieve what I am currently doing while reducing the number of calls to Mongo?
I'm currently hitting Mongo at about 50-400 reads (specifically for pricing) per user search, which slows things down a bit when combined with other queries, and is downright inefficient. I am considering a number of options:
- Creating a bucket (maybe an object in the
global
namespace in Node) where I cache all the pricing info by_id
. Each time I need to find pricing for aService
, I go through that bucket, if the pricing has already been cached in the bucket, I do my calculations and return the price, else I fetch the document from Mongo and add it to the bucket for current and future use.
I am worried this might introduce problems such as memory leaks. Ideally the bucket should be available to all the users, not like a cookie that belongs only to one user (my understanding of cookies might be wrong here). I don't yet know if it's possible to expose an object globally for all users to have access to it.
- I have seen a number of those 'in memory implementations' of Mongo (can't find any at the moment), which emulate MongoDB but using LocalStorage as a data store. I am considering forking one of them and tweaking them for server-side functionality.
I don't know if this could work, and it seems to be likely to take me some time to do.
- Store the data as a JSON file (or a number of files, I can work out a way of splitting the files). I can then read the JSON files with Node by
require()
ing them. This is currently my preferred option, as I can just use thefs
API to handle the periodic updating of the JSON files.
I am concerned about the size of the JSON files, mainly that if they're sufficiently large, they might slow Node down, but I'm considering this as I'll be saving 50 - 400 hits on the DB. I am also assuming that I won't have a single JSON file loaded n
times (n
being the number of users on the service at a point in time)
Considering the above 3 options, which has a good chance of working? Some of the options might be impossible, so I can scratch them off my list, there might be others which I haven't yet considered.
Lastly, my disclaimer: I know that this could have been implemented easily in SQL using JOIN statements, and no I am not using a hammer for something that needs a screwdriver. Although for some parts of my project SQL looks like a clear win, for a large part of the data that I store, Mongo is better. Plus reinventing the wheel in the name of experience helps.
Thanks