0

I have implemented several counters in my project using the new increment API. There are two functions whose implementation is a simple one line:

exports.MY_FUNC = functions.firestore.document('/PATH_TO_DOC/{id}').onCreate((snapshot, context) => {
    return admin.firestore().collection('COUNTERS').doc('DOC').update({COUNTER: admin.firestore.FieldValue.increment(1)});
});

(everything in CAPS is a generic replacement for the actual values which I have).

This function works perfectly well. However, each day, there are instances when it fails with "Error: memory limit exceeded. Function invocation was interrupted." It puzzles me as it works most of the time correctly.

I do know how to increase the memory limit, and I am going to increase it to 512MB right now. However, I don't like the unpredictability of this setup. Undoubtedly, the same function operating a counter on the same doc should either work correctly with 256MB or should fail every time.

I am also quite stunned that an operation which fits into one line may require more than 256MB. I have other functions which perform 3-4 different steps (reads, writes) and they work correctly with 256MB.

Given the unpredictability, I now worry that 512MB may also not be enough. Unfortunately, it doesn't seem that docs mention the memory requirements for increment API, so it looks like my only option is to deploy and see if the new memory limit is sufficient.

I would appreciate if anyone had any suggestions on any points below:

  1. Is there a way to enforce consistent behaviour? I want the function to either fail every time if 256MB is not enough or succeed every time if it is enough.

  2. How can developers estimate in advance the memory requirement of a particular function? I honestly do not think that it is OK to have a working function with 256MB during debugging only to find out that it may exceed memory limit in production.

Thanks for any input.

Andriy Gordiychuk
  • 5,818
  • 1
  • 20
  • 52
  • 2
    It seems very unlikely that your single line of code is using up that much memory. Which in turn makes it most likely that it's the context that the code runs in that is using the memory, most likely the size of the Admin SDK and any other modules you're importing. Be sure to only import the Firestore module of that, and be sure to only include NPM modules that this code actually needs (which is very few). – Frank van Puffelen Jul 07 '19 at 13:26
  • @FrankvanPuffelen thank you very much for your great advice. I will now remove unneeded packages. However, as I have multiple functions in my index.js, how could I fine tune which function requires which package? I assume that as my `require` declarations are before my functions' implementations, each function loads the same node modules, even if it doesn't need them. – Andriy Gordiychuk Jul 07 '19 at 13:39
  • 1
    One trick is to put the `require`s *inside* the function that needs them. This is usually done to speed up cold starts, but should also reduce memory usage. See an example here: https://stackoverflow.com/a/52056785 and here: https://stackoverflow.com/a/46572787 – Frank van Puffelen Jul 07 '19 at 13:44
  • @FrankvanPuffelen so far so good. However, I have now decided to update all of my functions to ensure that they do not load unnecessary packages. And in doing so, I also upgraded most of them from Node 6 to Node 8 runtime. One of my functions used to consume 10MB of memory. Now, it consumes 220MB. It is still below the limit of 256MB, but I do not understand why there is such a big increase? – Andriy Gordiychuk Jul 07 '19 at 14:03
  • 1
    Please post the entire file. the code in they entire file running is the function environment. Do the memory increase over time? if so it means you are doing something wrong. I would start by extracting the `admin.firestore()` to a const like here - https://github.com/firebase/functions-samples/blob/master/presence-firestore/functions/index.js#L25 – Boaz Jul 07 '19 at 14:19
  • @Boaz I managed to further reduce the memory consumption by moving initialization of a couple of packages inside the functions which use them. The last question I have is whether I can do the same for the `admin = require('firebase-admin');`? I have a couple of functions which do not interact with firestore, so I would imagine that I can avoid using admin. However, when I tried to do it, I got an error, so I wonder whether `admin` is required to be a top-level declaration for functions to work. – Andriy Gordiychuk Jul 07 '19 at 14:43
  • I don't think that's required. But without seeing an [MCVE](http://stackoverflow.com/help/mcve), it's hard to be certain what problem you're hitting. – Frank van Puffelen Jul 07 '19 at 19:08

0 Answers0