28

I am using Java.net at one of my project. and I wrote a App Server that gets inputStream from a client. But some times my (buffered)InputStream can not get all of OutputStream that client sent to my server. How can I write a wait or some thing like that, that my InputStream gets all of the OutputStream of client?

(My InputStream is not a String)

private Socket clientSocket;
private ServerSocket server;
private BufferedOutputStream outputS;
private BufferedInputStream inputS;
private InputStream inBS;
private OutputStream outBS;

server = new ServerSocket(30501, 100);
clientSocket = server.accept();

public void getStreamFromClient()  {
    try {
        outBS = clientSocket.getOutputStream();
        outputS = new BufferedOutputStream( outBS);
        outputS.flush();

        inBS = clientSocket.getInputStream();
        inputS = new BufferedInputStream( inBS );

    } catch (Exception e) {
        e.printStackTrace();
    }
}

Thanks.

A R
  • 375
  • 1
  • 4
  • 11

3 Answers3

63

The problem you have is related to TCP streaming nature.

The fact that you sent 100 Bytes (for example) from the server doesn't mean you will read 100 Bytes in the client the first time you read. Maybe the bytes sent from the server arrive in several TCP segments to the client.

You need to implement a loop in which you read until the whole message was received. Let me provide an example with DataInputStream instead of BufferedinputStream. Something very simple to give you just an example.

Let's suppose you know beforehand the server is to send 100 Bytes of data.

In client you need to write:

byte[] messageByte = new byte[1000];
boolean end = false;
String dataString = "";

try 
{
    DataInputStream in = new DataInputStream(clientSocket.getInputStream());

    while(!end)
    {
        int bytesRead = in.read(messageByte);
        dataString += new String(messageByte, 0, bytesRead);
        if (dataString.length == 100)
        {
            end = true;
        }
    }
    System.out.println("MESSAGE: " + dataString);
}
catch (Exception e)
{
    e.printStackTrace();
}

Now, typically the data size sent by one node (the server here) is not known beforehand. Then you need to define your own small protocol for the communication between server and client (or any two nodes) communicating with TCP.

The most common and simple is to define TLV: Type, Length, Value. So you define that every message sent form server to client comes with:

  • 1 Byte indicating type (For example, it could also be 2 or whatever).
  • 1 Byte (or whatever) for length of message
  • N Bytes for the value (N is indicated in length).

So you know you have to receive a minimum of 2 Bytes and with the second Byte you know how many following Bytes you need to read.

This is just a suggestion of a possible protocol. You could also get rid of "Type".

So it would be something like:

byte[] messageByte = new byte[1000];
boolean end = false;
String dataString = "";

try 
{
    DataInputStream in = new DataInputStream(clientSocket.getInputStream());
    int bytesRead = 0;

    messageByte[0] = in.readByte();
    messageByte[1] = in.readByte();

    int bytesToRead = messageByte[1];

    while(!end)
    {
        bytesRead = in.read(messageByte);
        dataString += new String(messageByte, 0, bytesRead);
        if (dataString.length == bytesToRead )
        {
            end = true;
        }
    }
    System.out.println("MESSAGE: " + dataString);
}
catch (Exception e)
{
    e.printStackTrace();
}

The following code compiles and looks better. It assumes the first two bytes providing the length arrive in binary format, in network endianship (big endian). No focus on different encoding types for the rest of the message.

import java.nio.ByteBuffer;
import java.io.DataInputStream;
import java.net.ServerSocket;
import java.net.Socket;

class Test
{
    public static void main(String[] args)
    {
        byte[] messageByte = new byte[1000];
        boolean end = false;
        String dataString = "";

        try 
        {
            Socket clientSocket;
            ServerSocket server;

            server = new ServerSocket(30501, 100);
            clientSocket = server.accept();

            DataInputStream in = new DataInputStream(clientSocket.getInputStream());
            int bytesRead = 0;

            messageByte[0] = in.readByte();
            messageByte[1] = in.readByte();
            ByteBuffer byteBuffer = ByteBuffer.wrap(messageByte, 0, 2);

            int bytesToRead = byteBuffer.getShort();
            System.out.println("About to read " + bytesToRead + " octets");

            //The following code shows in detail how to read from a TCP socket

            while(!end)
            {
                bytesRead = in.read(messageByte);
                dataString += new String(messageByte, 0, bytesRead);
                if (dataString.length() == bytesToRead )
                {
                    end = true;
                }
            }

            //All the code in the loop can be replaced by these two lines
            //in.readFully(messageByte, 0, bytesToRead);
            //dataString = new String(messageByte, 0, bytesToRead);

            System.out.println("MESSAGE: " + dataString);
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}
Uri
  • 10,163
  • 5
  • 44
  • 77
rodolk
  • 4,969
  • 3
  • 22
  • 32
  • 1
    Thanks rodok. I had a similar problem in writing a HTTP server. I managed to solve the problem using 'Content-Length' header as a mechanism to wait for exact amount of data from client. – Sali Hoo Jan 15 '16 at 18:32
  • @SaliHoo, yes that's the way to do it: with HTTP you need to read until you find 'Content-Length' (then you read the rest with the number you obtained) but there are other two cases to consider: when 'Content-Length' is not present and you find two '\r\n' (content is 0) and when the message contains chunks. – rodolk Jan 15 '16 at 18:35
  • What if you don't know the `bytesToRead` - i.e. it could vary from message to message? I have this problem and it has been *very* hard to solve. – StephenBoesch Nov 06 '17 at 18:39
  • @javadba the bytesToRead comes in each message, in a field indicating the message length. When the client writes the message, it also writes the length at a fixed position in the message. It is explained in my response. Alternatively, the message has to give you other information that lets you derive the message. Finally, if you don't have this information you have two alternatives: 1-Fixed message 2-Message delimiter (this works but shouldn't be necessary). – rodolk Nov 06 '17 at 19:40
  • I see- you've added it in to make a custom protocol. yes that is what I had already resorted to including. The surprise is that *without* adding basically the socket comm does not work. This is not covered in any tutorial. – StephenBoesch Nov 06 '17 at 20:50
  • @rodolk Best answer! I am facing same issue here. Is there any solution for improving the byte read speed or do the reading async? My problem is that server is sending faster than client reads... – George Fandango Jun 14 '18 at 13:37
  • 1
    @GeorgedeLemos Most probably the problem is that the client is processing the message with the same thread it's reading (I suppose so). Whatever it is the client is taking too much time for additional processing. TCP has a mechanism for flow control, if the client is consuming data at a slower pace it arrives from server, at some point the TCP receive buffer will be filled and TCP will tell the server to stop sending with an advertised window wtih value 0 (zero). That should manage the situation. However your concern must be related to how slow the client is. Use a different thread to process. – rodolk Jun 16 '18 at 00:11
  • @rodolk made that: reading is parallelized in another thread. I continued analyzing and my problem is TCP connection related as you answered before. My solution would be client to reconnect newly after “message” is sent but due to it is a third-party software I can not make any changes about that. Thank you so much for your help! :) – George Fandango Jun 16 '18 at 08:46
  • How can i distinguish one message from one another if there is already more than one message in socket(think server busy and client send data really fast)? Predefined buffer size allows retrieving data more than first message size because there is no check for this. For example assume that first message size is 20 byte. Before server read the message we has sent another 30 byte message. After retrieving these two messages how server distinguish first message from second message. Server can read from inputstream 0 to 50 byte. In this scenario is it possible that second message data lost? – MstfAsan Aug 25 '18 at 14:34
  • @MstfAsan please read the code carefully. You'll see a technique here: the message should always have a fixed header. You know there are at least 2 Bytes at the beginning and, for example, the second Byte will specify the message length. So you know how many octets belong to the first message and this is what you are going to read next. The example code shows that. Then you'll begin to read the next msg. Of course, there is the possibility of getting out of sync if you have an error reading, then you may lose the limit between messages. – rodolk Aug 27 '18 at 01:36
  • i have really read your post carefully but even if we add the length of message in second byte read() tries to read as much as buffer size (1024 your buffer size). In this case read() can retrieve more than length of first message size then `if (dataString.length() == bytesToRead ) { end = true; }` never been hit. – MstfAsan Aug 27 '18 at 06:26
  • How should I fit the packages received out of order in this schema? Or, I don't even need to think about it as TCP handles that for me? For instance, if I ask for 4096 bytes via a `read` call, would that mean a single and blocking request for that TCP socket? I mean will the socket make sure all the packages received (even out of order) and reassembled on the client side connection, then hand over the data to the application? – stdout Mar 27 '20 at 13:25
  • 1
    @stdout ordering is handled by TCP. It delivers data in order to the application layer. You don't need to worry about it in the application layer. There are other complicated scenarios with asynchronous communication but I suppose you're not referring to it. – rodolk Mar 27 '20 at 19:34
  • I was just trying to blend your answer with how, for instance, an input stream functions. From my understanding, I see each read call as a separate blocking request over a same http connection and the underlying TCP protocol handles it’s mess (lost or out of order packets) pretty much as you said. – stdout Mar 27 '20 at 22:09
  • 1
    @stdout, it's difficult to understand your comment. A TCP connection can be for HTTP or for any other application layer protocol. In HTTP the goal of the server is to read a complete HTTP request. For that, you have to apply what is explained in my response. You need to read everything until you find the HTTP message length in the HTTP header, in whatever position it is. Then you know the total amount of Bytes you need to read for the complete request. – rodolk Mar 30 '20 at 16:26
  • @rodolk that was almost what I was trying to say. But also I'm trying to figure out, for instance, what happens if I send more than one (HTTP/2) GET requests over the same TCP connection (for instance when you load a new webpage on the browser) and a TCP frame of the first request gets lost? Then this would block subsequent requests regardless some of them are even completed on the receiving side. I'm wondering how TCP knows about this and block since it doesn't know anything about the upper level protocol? – stdout Apr 01 '20 at 10:12
  • For future readers, this problem was also reported to java https://bugs.openjdk.java.net/browse/JDK-4406875 – Umar Tahir Dec 02 '20 at 15:26
  • @UmarTahir the one you pointed out is really a bug that generates a different problem, loss of data in case of an IOException. In this case, the discussion is about how to read the data given how TCP works. – rodolk Dec 07 '20 at 21:01
  • Yes, the discussion is about TCP. If you read all the details of the bug it is highlighting the problem in Java Socket. – Umar Tahir Jan 06 '21 at 13:56
1
int c;
    String raw = "";
    do {
        c = inputstream.read();
        raw+=(char)c;
    } while(inputstream.available()>0);

InputStream.available() shows the available bytes only after one byte is read, hence do .. while

  • In network communication, knowledge of length cannot be interrogated by `.available()` but instead, only by a header – eigenfield Oct 19 '20 at 10:20
0

You can read your BufferedInputStream like this. It will read data till it reaches end of stream which is indicated by -1.

inputS = new BufferedInputStream(inBS);
byte[] buffer = new byte[1024];    //If you handle larger data use a bigger buffer size
int read;
while((read = inputS.read(buffer)) != -1) {
    System.out.println(read);
    // Your code to handle the data
}
Code.me
  • 253
  • 2
  • 13
  • 1
    I was just showing how to read the entire input stream. println() is just a place holder here. The OP has not asked how to get the read count! If he understands how to read the entire stream, he will understand how to get the read count. – Code.me Nov 08 '13 at 04:29
  • OP is using InputStream. The while loop will hang forever unless he sets a timeout, like: socket.setSoTimeout(1000); – SoloPilot Sep 25 '14 at 21:19
  • It's *this code* that needs to use the read count. Your 'handle the data' handwaving doesn't even mention it. – user207421 Feb 11 '15 at 21:58