18

Currently I'm hitting a hard limit of 130688 bytes. If I try and send anything larger in one message I get a ENOBUFS error.

I have checked the net.core.rmem_default, net.core.wmem_default, net.core.rmem_max, net.core.wmem_max, and net.unix.max_dgram_qlen sysctl options and increased them all but they have no effect because these deal with the total buffer size not the message size.

I have also set the SO_SNDBUF and SO_RCVBUF socket options, but this has the same issue as above. The default socket buffer size are set based on the _default socket options anyways.

I've looked at the kernel source where ENOBUFS is returned in the socket stack, but it wasn't clear to me where it was coming from. The only places that seem to return this error have to do with not being able to allocate memory.

Is the max size actually 130688? If not can this be changed without recompiling the kernel?

red0ct
  • 4,231
  • 3
  • 14
  • 37
Jaime
  • 1,052
  • 2
  • 12
  • 26
  • 2
    That is a huge datagram. In my opinion, by the time that you have a datagram that large, you may as well have used TCP. – Jon Trauntvein Jan 18 '11 at 21:37
  • 1
    Yea, that doesn't help. As I stated in the post, it won't let you send a message over 130688 regardless of your wmem settings. I have them over 32MB and have tried many combinations below that. – Jaime Jan 18 '11 at 22:04
  • 1
    Just to add to that. Its a misconception that the send buffers and receive buffers are for single messages. The buffer is the total kernel buffer for all messages. The wmem and qlen sysctl options actually will effect how and when send blocks. As the send buffer fills up (assuming no one receives), when the total bytes in the buffer would go beyond the buffer size or the total count will go beyond the qlen, send will block. – Jaime Jan 18 '11 at 22:15
  • I get your point (and the question) better. Redacted confusing comment and upvoted; will explore around as time permits, as I'm interested in the answer as well. – JB. Jan 18 '11 at 22:40
  • That's 128kB minus some overhead (384 bytes). That sounds *entirely reasonable* to me as a maximum size of a message. If you were going over an external interface, the limit would be much lower. – Donal Fellows Jan 19 '11 at 13:45
  • 2
    I agree that its possible this is the hard limit. Just trying to find some proof and maybe some reasoning behind it. – Jaime Jan 19 '11 at 13:49

1 Answers1

18

AF_UNIX SOCK_DATAGRAM/SOCK_SEQPACKET datagrams need contiguous memory. Contiguous physical memory is hard to find, and the allocation fails, logging something similar to this on the kernel log:

udgc: page allocation failure. order:7, mode:0x44d0
[...snip...]
DMA: 185*4kB 69*8kB 34*16kB 27*32kB 11*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3788kB
Normal: 13*4kB 6*8kB 100*16kB 62*32kB 24*64kB 10*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 7012kB
[...snip...]

unix_dgram_sendmsg() calls sock_alloc_send_skb() lxr1, which calls sock_alloc_send_pskb() with data_len = 0 and header_len = size of datagram lxr2. sock_alloc_send_pskb() allocates header_len from "normal" skbuff buffer space, and data_len from scatter/gather pages lxr3. So, it looks like AF_UNIX sockets don't support scatter/gather on current Linux.

ninjalj
  • 39,486
  • 8
  • 94
  • 141
  • 2
    Excellent work. This is essentially what I found in my traces, but you have provided actual reason. I wonder why datagrams would have this limit but not streams? – Jaime Jan 28 '11 at 13:18
  • 2
    SOCK_STREAM sockets don't preserve message boundaries. – ninjalj Jan 28 '11 at 18:50
  • See also expanded answer at http://stackoverflow.com/questions/21856517/whats-the-practical-limit-on-the-size-of-single-packet-transmitted-over-domain – Jonathan Ben-Avraham Aug 22 '14 at 08:53
  • SOCK_STREAM sockets most certainly do preserve message boundaries. Just place each message into a separate stream: create a new socket, connect it, write the message, shut it down in the sending direction, receive the reply, and disconnect. Then repeat for the next message. That is to say, connections support framing because they have clearly defined "start of stream" and "end of stream" signaling; they just don't support framing within a stream. Just like SOCK_DGRAM sockets don't support framing within a datagram. – Kaz Mar 07 '16 at 15:54
  • 3
    @Kaz, so, they don't, and you have to fake it. – lvella Sep 10 '19 at 13:15