Short version:
I'm trying to segment a stream of bytes that may represent anything (for example, a TCP stream) and send them over multiple (unreliable) links to a receiving system under my control. There, I want to reorder these segments into the order the sending system first received them. See the second ascii image for an example. The resulting tunnel doesn't need to be as reliable as TCP (UDP-ish is fine), but it should handle the fact that multiple inner links are being used (which is not trivial).
Using a header that contains a size
and sequence number
field would solve all problems, but only in an perfect world. If one of the links misbehave and the peers get out of sync (meaning, the receiver will read bytes that it thinks should constitute a header, but don't) some mechanism of resyncing is needed. In regard to UDP, I suppose the major network stacks parse the complete UDP header and make an educated guess on whether it got out of sync, but are there good ways of getting back on track after such failure?
Longer version, use if more details/context is needed:
I'm experimenting with creating a single tunnel on top of several distinct links/tunnels. These links may be unreliable and it doesn't matter how they have been set up: sockets, tun, tap, what have you. The tunnel that I'm creating doesn't care. What it does care about, obviously, is that its customers get a good service, similar to what you would expect from UDP.
Current version:
I've implemented such a tool on a single, reliable link, as a first version (simply put, I ported the source referenced here to C++):
+------\ TCP /------+
tun socket <---> socket tun
+------/ \------+
client server
How this works is simple: the client reads bytes from its tun device and writes them into the socket. Both are simply implemented as performing a Linux read and write on file descriptors. The server does the reverse for receiving. This works full-duplex. Noteworthy here is:
- The client and server don't care about what the bytes they operate on represent, they just move them over to the other side. I have to assume that the Linux networking stack is doing smart stuff to understand what the bytes it gets mean, even in case of weird fragmentation.
- The two endpoints implement a very simply protocol, where each bunch of data that is written into the socket is preceded with the amount of bytes that are to be read (again, all this comes from the url above). This is how the endpoints keep track of things (it's only for improving performance, as far as I can see)
Next version:
Now, instead of the single stream socket, these endpoints will have to talk to each other using multiple links, like:
+----------\ /----------+
| .-- tun1A <---> tun1B --. |
tun0A ---- tapA <---> tapB ---- tun0B
| '-- socA <---> socB --' |
+----------/ \----------+
As there's no easy way of telling whether an endpoint has read a full TCP segment on its tun0 interface, I'm planning to keep regarding everything that comes in on the tun devices simply as a byte stream. I want to send it to the other side using any of the three links, with no regard whatsoever to what the original byte stream represented. This way, a bunch of bytes that originally represented a single TCP segment may be sent to the other side using 4 separate packages. These packages need to be reordered at the receiving end, which is no problem in itself: just add a second field to the one that I already have: a sequence number
, right?.
But, what if the underlying link is unreliable (which it probably is)? If the packages simply get a bit swapped in their payload, I assume, for now, that the networking stack at the receiving side is capable of coping with it. But what if the header gets broken? The endpoint might learn a wrong number of bytes that it should read and the two endpoints go out of sync with no apparent fix.
I've looked at implementing it using acks and timeouts of sorts, but I've not given up hope on a simpler and more elegant approach (remember that the end result doesn't need to be reliable). Some protocols use a reliable out-of-band channel (like TCP) for managing the payload tunnel, but the goal is to purely use the links. Also, even though Linux offers all kinds of cool networking tools (iproute2, netfilter, etc), all of this should be implemented in C/C++ (small concessions may be made here if a little use of those offers a good solution though).
Right now, I'm short on ideas and I'm hoping anyone could propose some other approaches to this problem! If more info is needed do ask, I'm happy to write more :)
*Originally put this question on networkengineering.stackexchange.
Edit 1
I'm thinking about wrapping every single "inner link" as shown in the image above with its own datagram socket wrapper. This socket would (hopefully) guarantee at least that I receive full datagrams so that I don't have to worry about the link breaking in case of truncated messages. It does add quite some overhead though.