7

We are using Apache Thrift to exchange messages between two systems. In one of the message we are exchanging a list (c++) which can become huge in size. Can you please let me know what is the maximum message size we can exchange using Apache Thrift?

JensG
  • 12,102
  • 4
  • 40
  • 51
user1732768
  • 81
  • 1
  • 4

1 Answers1

4

There is no defined "per se" limit (at least none that I am aware of). It mostly depends on how the data are held in memory, what load is on the server and how much resources are avilable. For the most part, contiguos blocks of memory (RAM) will very likely become the scarcest resource, so we should focus on that point.

The "how the data are held in memory" refers to the fact that for the sake of better throughput some transports (buffered, framed) tend to allocate more memory and larger blocks than others. Depending on the language's implementation this process may be implemented more or less efficient in terms of memory cost.

If you really plan to transfer large blocks of data, you should also look at other options like

  • chunking the data into blocks
  • sending/returning only an URL or LAN share through the service, instead of the whole data
JensG
  • 12,102
  • 4
  • 40
  • 51
  • Is there a canonical approach for "_chunking the data into blocks_" to send/receive data in the context of Thrift? Thanks! – blong May 10 '16 at 18:06
  • 1
    No, you have all freedom to implement it the way you need it ;-) – JensG May 10 '16 at 20:29
  • Hehe, fair enough! :) – blong May 10 '16 at 21:41
  • I still have the problem of sending large chunk ... 300kbytes .. If I send only 10kbyte, it's very slow when uploading. If I send large chunk, the service seems freezed the call and not go anywhere, not even reach to server code. I have no problem in 10kbytes but slow sending in local network. – LittleFunny Aug 04 '16 at 02:48
  • Hard to say ... I would probably try Wireshark or some tool like this to find out what's (not) going on here. – JensG Aug 04 '16 at 07:49