22

We have a TCP stream protocol where we prefix our data payload by the size. So the data can be properly decoded when received. Pretty standard stuff.

This is working fine for thousands of people. Unfortunately we have at least 4 reported cases of clients having connection problems, all in remote countries. A client in Russia has been able to help us run lots of tests and narrowed the problem down. If we send a packet where the prefix size is forced to be 0 then the entire packet will make it through. If the packet data starts with 1c the packet won't make it through.

I have two side by side Wireshark captures from his computer that show this:

Working
-------
Russia -> Toronto [SYN]
Toronto -> Russia [SYN, ACK]
Russia -> Toronto [ACK]
Russia -> Toronto [PSH,ACK] <- data is sent here
00000000000000001c0000000000000000000000000000000000000000000000
Toronto -> Rusion [PSH,ACK] <- server in toronto got the data, sent a reply!


Not-Working
--------
Russia -> Toronto [SYN]
Toronto -> Russia [SYN, ACK]
Russia -> Toronto [ACK]
Russia -> Toronto [PSH,ACK] <- data is sent here
1c000000000000001c0000000000000000000000000000000000000000000000
Russia -> Toronto [PSH,ACK] <- TCP Retransmission
Russia -> Toronto [PSH,ACK] <- TCP Retransmission
Russia -> Toronto [PSH,ACK] <- TCP Retransmission
Russia -> Toronto [PSH,ACK] <- TCP Retransmission

Server in Toronto never gets the packet from Russia!

The actual client and servers use IOCP but my test app uses C# TcpListener and TcpClient with NO custom options flags at all.

Not actual code
--------------
var client = new TcpClient()
client.Connect(host, port)
client.GetStream().Write()
client.GetStream().Read()

var listener = new TcpListener(port);
listener.Start();
var serverClient = listener.AcceptTcpClient();
serverClient.GetStream().Read()
serverClient.GetStream().Write()

Are there any further tests to recommend to get more information / solve this? My hunch was his hardware/drivers are corrupt but he claims to have no issues with any other application or internet in general.

Lightness Races in Orbit
  • 358,771
  • 68
  • 593
  • 989
Spish
  • 513
  • 3
  • 6
  • 7
    I would guess that there are some deep inspection devices in between which filter traffic. Often these devices are not very intelligent and just look at some pattern and ports. You might try to use alternative connections, e.g. try to use different ports or use a VPN tunnel. – Steffen Ullrich Apr 01 '14 at 08:47
  • We are using port 9001 for the server. Unfortunately VPN is not an option. The app in question is a game client, can't expect people to setup a VPN to play! – Spish Apr 01 '14 at 15:48
  • I did not mean to switch permanently to VPN or another port, but just do it to test, if the problem vanishes. If it vanishes it is probably a filtering middlebox in between which you either can try to work around or just give up with this client. If the problem persists you must look for other explanations. – Steffen Ullrich Apr 01 '14 at 15:55
  • That makes more sense, yes the test works over VPN. What sort of work arounds would you propose? We have tried a one different port already. I guess try more? – Spish Apr 01 '14 at 17:30
  • 3
    This looks then like a deep inspection middlebox interfering with the traffic, often used to hamper P2P networks. If you cannot get another path (e.g. VPN) you can try to trick the box into thinking that you speak some "normal" protocol, e.g. make the initial handshake look like HTTP or SSL and use ports usually associated with these protocols. Other then that there is not much you can do :( – Steffen Ullrich Apr 01 '14 at 18:56
  • 6
    if @SteffenUllrich suguestion dosn't work (and i feel dirty for making this suggestion) try padding the payload with a zero. – Robert Apr 02 '14 at 09:08
  • What happens if you compose and send a packet using "TCP Packet Generator" or "Packet Sender" or similar app? – Alexandr Nikitin Apr 03 '14 at 18:38
  • 3
    – lorenzog Apr 04 '14 at 08:47
  • 2
    It sounds like 1c prefix is detected as some specific unwanted stream of data. Did you try to switch to HTTPS to skip this kind of testing? – Artem Apr 04 '14 at 17:36
  • 2
    Just as an update. I changed our packet data protocol to pad the payload with a zero byte. Fixed... I feel dirty using Robert's solution but well it works everywhere now. – Spish Apr 04 '14 at 23:08

1 Answers1

2

Could it be that one of the end links is over a mobile carrier?

I don't have the data any more, but since we're in the realm of speculations I remember a similar problem with an Italian mobile carrier a while ago: apparently, a certain sequence of bits over a data connection would drop the carrier. Vaguely similar to the +++ATH0 old 'ping' trick.

Could you try sending a similar sequence (1c0000000....) through another medium, say a netcat stream?

lorenzog
  • 3,037
  • 3
  • 26
  • 47