TCP socket sharding in .NET

Question

I'm trying to write a high-performance HTTP server in C# without using third-party libraries, and during my readings, I found this post on nginx blog. What mentioned there is using SO_REUESEPORT but that option is not available in .NET sockets and also Windows, so I searched for an alternative to this option and I've read that SO_REUSEADDR which is available in both Windows and .NET is the option that I have looking for.

I have tried to implement that by creating 4 sockets and enabling reuse-address before binding the sockets and start listening, but what happens after is only one of the sockets accepts connections even though its thread goes busy.

static readonly byte[] Response = Encoding.ASCII.GetBytes("HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r");

static int s_ListenerIndex = 0;

static void Main(string[] args)
{
    var listenAddress = new IPEndPoint(IPAddress.Any, 26000);

    var sockets = new Socket[4];

    Console.WriteLine("Creating sockets");

    for (int i = 0; i < sockets.Length; i++)
    {
        sockets[i] = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
        sockets[i].SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        sockets[i].Bind(listenAddress);
    }

    Console.WriteLine("Starting listener threads");

    for (int i = 0; i < sockets.Length; i++)
    {
        var socket = sockets[i];
        CreateThread(() => ListenSocket(socket));
    }

    Thread.Sleep(Timeout.Infinite);
}

static void ListenSocket(Socket socket)
{
    var listenerId = Interlocked.Increment(ref s_ListenerIndex);
    socket.Listen(100);
    Console.WriteLine($"Socket #{listenerId} is listening");

    while (true)
    {
        Socket client = socket.Accept();
        Console.WriteLine($"Socket #{listenerId} accepted connection");
        client.Send(Response);
        client.Close();
    }
}

static void CreateThread(ThreadStart threadStart)
{
    var thread = new Thread(threadStart);
    thread.IsBackground = true;
    thread.Start();
}

Am I doing something wrong or the whole trick is not possible in Windows and .NET?

According to MS (see [here](https://docs.microsoft.com/en-us/windows/win32/winsock/using-so-reuseaddr-and-so-exclusiveaddruse) for example), if using SO_REUSEADDRESS "the behavior for all sockets bound ... is indeterminate". In other words, non-deterministic. Its only legitimate use is to avoid TIME_WAIT problems while restarting a server. I´m curious of what you´re trying to achieve, btw. Network communication is serial and a single server thread is more than capable of handling incoming connections at full speed... The processing of each connection is then handled as needed. — C. Gonzalez, Jan 05 '20 at 15:20
@C.Gonzalez My benchmarks have shown that https://github.com/valyala/fasthttp accepts connections faster than a normal socket that accepts connections using blocking accept() method plus a static response like the example I've provided. That means what I'm looking for is possible on Windows or there are alternatives to it. — moien, Jan 06 '20 at 14:56
Your listening socket currently waits for your server to handle the request, before being able to accept the next incoming connection. You could process the `client` socket in another thread and get to `socket.Accept();` as soon as possible. Have you considered the asynchronous calls? — C.Evenhuis, Jan 11 '20 at 08:55
Its hard to figure out exactly what you want to achieve. If you "just want SO_REUSEADDRESS" to work, you should let the workers take a considerable amount of time to process a request so its listener isn't available earlier than the next connection attempt. — C.Evenhuis, Jan 11 '20 at 11:54
But if you plan on creating a stable and performant server, I wouldn't worry about speed that much in this stage of development. I'm pretty sure the cost of a lock (they speek about in the link you provided) isn't that much when workers are doing _actual work_. — C.Evenhuis, Jan 11 '20 at 12:07

score 1 · Accepted Answer · answered Jan 11 '20 at 08:41

On Windows, the SO_REUSEADDR option has no use cases for an HTTP server. It's only real use case is for multicast sockets where data is delivered to all of the sockets bound on the same port.

I suggest this SO post for a indepth discussion. How do SO_REUSEADDR and SO_REUSEPORT differ?

Under Windows with IOCP you don't need the Linux/Unix equivalent which can provide load balancing across multiple threads. IOCP takes care of this for you. The implementation is very different but achieves the same highly scalable result.

TCP socket sharding in .NET

1 Answers1