3

I'm evaluating using NSQ, http://nsq.io/, for a specific project. The idea is to setup a data pipeline where each step is a job, and where the state ideally will be located in the message body.

Which got me to think about a potential maximal message size. I don't manage to find any documentation on the subject. Can it be any size? I guess it will affect performance if messages are big enough to not fit in memory.

Niklas9
  • 7,378
  • 7
  • 31
  • 54

2 Answers2

4

As Aldo mentioned, the default maximum message size is around 1 megabyte.

This is easy enough to change with the max-msg-size switch for the NSQ daemon. Here is an example of how to use it:

nsqd --lookupd-tcp-address=127.0.0.1:4160 -max-msg-size=2097152

This tells the NSQ daemon (nsqd) that max message size should be 2 megabytes instead of the default.

Using this setting, you could indeed have a message that would fill up your memory, as you mentioned in a comment.

SunSparc
  • 1,672
  • 2
  • 20
  • 45
2

as far as I can tell from the documentation, it looks like it's 1MB by default.

I know it sounds like a lot but I managed to hit this limit.

What I'm doing now is to send an NSQ message with the minimum amount of data regarding the event, and get the full data on the other end when I handle it.

Aldo 'xoen' Giambelluca
  • 9,881
  • 7
  • 28
  • 39
  • Cool! Thanks a lot. It's a setting though, so I guess you could potentially have a big of a size that would fit in memory ? And what do you mean by "on the other end", a state in another db ? – Niklas9 Dec 01 '15 at 18:23
  • By "on the other end" I mean that I'm getting more details about the event in the NSQ message when I'm consuming it ("the other end" :P). – Aldo 'xoen' Giambelluca Dec 14 '15 at 13:20