Android to PHP: epipe Broken Pipe / stream closed posting chunked multipart form-data using httpurlconnection

Question

I'm having a problem uploading video files from android app using HttpUrlConnection to server and handling it using PHP. This is done successfully with video files of up to 12mb in size by sending a multipart post request but fails for a file of 16mb with out of memory error.

I tried to solve the out of memory error via chunking using setChunkedStreamingMode(1024) or using setFixedLengthStreamingMode(contentLength). However this results in errors like epipe (broken pipe) and stream closed. I don't know how to handle that.

What I Tried

I first tried using Kevin Sawicki's http-request which resulted in out of memory error for large files. When I implemented chunking or setFixedLengthStreamingMode, that results in epipe broken pipe error.

Android Code in AsyncTask:

/**
 * Working code for uploading photos and small videos (tested with 12mb) from gallery and camera
 */
// set post data
request = HttpRequest.post(postUrl);
// request.chunk(1024*1024); // epipe (broken pipe) error if uncommented
request.part("user_id", userId);
request.part("question_title", title);
request.part("question_subject", subject);
request.part("question_level", level);
// add photo or video
if (mediaType == PHOTO) {
    request.part("file", photoFile.getName(), photoFile);
}
else if (mediaType == VIDEO) {
    request.part("file", videoFile.getName(), "video/mp4", videoFile);
}

if (request.ok()) {
    try {
        return new JSONObject(request.body().toString());
    } catch (HttpRequestException e) {
        e.printStackTrace();
    } catch (JSONException e) {
        e.printStackTrace();
    }
}

LogCat:

10-24 16:26:59.365: E/AndroidRuntime(19014): FATAL EXCEPTION: AsyncTask #4
10-24 16:26:59.365: E/AndroidRuntime(19014): java.lang.RuntimeException: An error occured while executing doInBackground()
10-24 16:26:59.365: E/AndroidRuntime(19014):    at android.os.AsyncTask$3.done(AsyncTask.java:299)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:352)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.FutureTask.setException(FutureTask.java:219)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.FutureTask.run(FutureTask.java:239)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.lang.Thread.run(Thread.java:841)
10-24 16:26:59.365: E/AndroidRuntime(19014): Caused by: com.example.askacademy.HttpRequest$HttpRequestException: java.net.SocketException: sendto failed: EPIPE (Broken pipe)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest$Operation.call(HttpRequest.java:689)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest.copy(HttpRequest.java:2586)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest.part(HttpRequest.java:2899)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest.part(HttpRequest.java:2866)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:447)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:1)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at android.os.AsyncTask$2.call(AsyncTask.java:287)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.util.concurrent.FutureTask.run(FutureTask.java:234)
10-24 16:26:59.365: E/AndroidRuntime(19014):    ... 4 more
10-24 16:26:59.365: E/AndroidRuntime(19014): Caused by: java.net.SocketException: sendto failed: EPIPE (Broken pipe)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.IoBridge.maybeThrowAfterSendto(IoBridge.java:499)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.IoBridge.sendto(IoBridge.java:468)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.net.PlainSocketImpl.write(PlainSocketImpl.java:507)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.net.PlainSocketImpl.access$100(PlainSocketImpl.java:46)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.net.PlainSocketImpl$PlainSocketOutputStream.write(PlainSocketImpl.java:269)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.net.http.ChunkedOutputStream.writeHex(ChunkedOutputStream.java:102)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.net.http.ChunkedOutputStream.writeBufferedChunkToSocket(ChunkedOutputStream.java:128)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.net.http.ChunkedOutputStream.write(ChunkedOutputStream.java:77)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:131)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest$8.run(HttpRequest.java:2580)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest$8.run(HttpRequest.java:1)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at com.example.askacademy.HttpRequest$Operation.call(HttpRequest.java:683)
10-24 16:26:59.365: E/AndroidRuntime(19014):    ... 11 more
10-24 16:26:59.365: E/AndroidRuntime(19014): Caused by: libcore.io.ErrnoException: sendto failed: EPIPE (Broken pipe)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.Posix.sendtoBytes(Native Method)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.Posix.sendto(Posix.java:155)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.BlockGuardOs.sendto(BlockGuardOs.java:177)
10-24 16:26:59.365: E/AndroidRuntime(19014):    at libcore.io.IoBridge.sendto(IoBridge.java:466)
10-24 16:26:59.365: E/AndroidRuntime(19014):    ... 21 more

I then tried to send the file only without a library, code from StackOverflow, which results in stream closed error.

Android Code:

HttpURLConnection conn = null;
DataOutputStream dos = null;
DataInputStream inStream = null;
String lineEnd = "\r\n";
String twoHyphens = "--";
String boundary = "***************************************************";
int bytesRead, bytesAvailable, bufferSize;
byte[] buffer;
int maxBufferSize = 212144; // 1024*1024 = 1MB.  212144 is a quarter MB.
FileInputStream fileInputStream = null;

try {
    fileInputStream = new FileInputStream(videoFile);
    // open a URL connection to the Servlet
    URL url = new URL(postUrl);
    // Open a HTTP connection to the URL
    conn = (HttpURLConnection) url.openConnection();
    // Allow Inputs
    conn.setDoInput(true);
    // Allow Outputs
    conn.setDoOutput(true);
    // Send in chunks (to avoid out of memory error)
    conn.setChunkedStreamingMode(maxBufferSize); // results in stream closed error
    // Don't use a cached copy.
    conn.setUseCaches(false);
    // Use a post method.
    conn.setRequestMethod("POST"); 
    conn.setRequestProperty("Connection", "Keep-Alive");
    conn.setRequestProperty("Content-Type", "multipart/form-data;boundary="
        + boundary);
    conn.setReadTimeout(200000); // 200 seconds...
    dos = new DataOutputStream(conn.getOutputStream());
    dos.writeBytes(twoHyphens + boundary + lineEnd);
    dos.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\""
        + videoFile.getName() + "\"" + lineEnd);
    dos.writeBytes(lineEnd);
    // create a buffer of maximum size
    bytesAvailable = fileInputStream.available();
    bufferSize = Math.min(bytesAvailable, maxBufferSize);
    buffer = new byte[bufferSize];
    // read file and write it into form...
    bytesRead = fileInputStream.read(buffer, 0, bufferSize);
    while (bytesRead > 0)
    {
      try {
        dos.write(buffer, 0, bufferSize);          
      } catch (OutOfMemoryError oome) {
        Log.e(DEBUG_TAG, "Out of memory error caught...");
        oome.printStackTrace();
        fileInputStream.close();
        throw new Exception("Out Of Memory!");
      }
      bytesAvailable = fileInputStream.available();
      bufferSize = Math.min(bytesAvailable, maxBufferSize);
      bytesRead = fileInputStream.read(buffer, 0, bufferSize);
      // send multipart form data necesssary after file data...
      dos.writeBytes(lineEnd);
      dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);
      fileInputStream.close();
      dos.flush();
      dos.close();

      // close streams
      Log.d(DEBUG_TAG, "Backup file written to server successfully...");
    }
} catch (Exception e) {
    // TODO: handle exception
    e.printStackTrace();
}

Logcat

10-24 16:29:29.070: I/dalvikvm-heap(19832): Grow heap (frag case) to 23.474MB for 4147216-byte allocation
10-24 16:29:29.090: D/dalvikvm(19832): GC_FOR_ALLOC freed 1K, 20% free 22595K/28144K, paused 20ms, total 20ms
10-24 16:29:29.100: W/CursorWrapperInner(19832): Cursor finalized without prior close()
10-24 16:29:30.700: D/ProgressBar(19832): setProgress = 0
10-24 16:29:30.700: D/ProgressBar(19832): setProgress = 0, fromUser = false
10-24 16:29:30.700: D/ProgressBar(19832): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 10000
10-24 16:29:30.770: D/CreateQuestion(19832): Backup file written to server successfully...
10-24 16:29:30.770: W/System.err(19832): java.io.IOException: stream closed
10-24 16:29:30.775: W/System.err(19832):    at libcore.net.http.AbstractHttpOutputStream.checkNotClosed(AbstractHttpOutputStream.java:37)
10-24 16:29:30.775: W/System.err(19832):    at libcore.net.http.ChunkedOutputStream.write(ChunkedOutputStream.java:65)
10-24 16:29:30.775: W/System.err(19832):    at java.io.DataOutputStream.write(DataOutputStream.java:98)
10-24 16:29:30.775: W/System.err(19832):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:405)
10-24 16:29:30.775: W/System.err(19832):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:1)
10-24 16:29:30.775: W/System.err(19832):    at android.os.AsyncTask$2.call(AsyncTask.java:287)
10-24 16:29:30.780: W/System.err(19832):    at java.util.concurrent.FutureTask.run(FutureTask.java:234)
10-24 16:29:30.780: W/System.err(19832):    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
10-24 16:29:30.780: W/System.err(19832):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
10-24 16:29:30.780: W/System.err(19832):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
10-24 16:29:30.780: W/System.err(19832):    at java.lang.Thread.run(Thread.java:841)
10-24 16:29:30.780: W/System.err(19832): org.json.JSONException: Unterminated object at character 52 of {"success":"false","error":"Error creating question"
10-24 16:29:30.780: W/System.err(19832):    at org.json.JSONTokener.syntaxError(JSONTokener.java:450)
10-24 16:29:30.780: W/System.err(19832):    at org.json.JSONTokener.readObject(JSONTokener.java:394)
10-24 16:29:30.780: W/System.err(19832):    at org.json.JSONTokener.nextValue(JSONTokener.java:100)
10-24 16:29:30.780: W/System.err(19832):    at org.json.JSONObject.<init>(JSONObject.java:154)
10-24 16:29:30.780: W/System.err(19832):    at org.json.JSONObject.<init>(JSONObject.java:171)
10-24 16:29:30.780: W/System.err(19832):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:461)
10-24 16:29:30.780: W/System.err(19832):    at com.example.askacademy.CreateQuestionActivity$createQuestionTask.doInBackground(CreateQuestionActivity.java:1)
10-24 16:29:30.780: W/System.err(19832):    at android.os.AsyncTask$2.call(AsyncTask.java:287)
10-24 16:29:30.780: W/System.err(19832):    at java.util.concurrent.FutureTask.run(FutureTask.java:234)
10-24 16:29:30.780: W/System.err(19832):    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
10-24 16:29:30.780: W/System.err(19832):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
10-24 16:29:30.785: W/System.err(19832):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
10-24 16:29:30.785: W/System.err(19832):    at java.lang.Thread.run(Thread.java:841)

Questions So my questions are: How do I solve the problem of connection/stream closing when chunking?

If the connection problem is due to server side issues, how do I find out what caused it and solve them? Note that the upload_max_filesize is 512M so it's not the size that's a problem. Also, 12mb works and 16mb fails.

I'm not sure how to process the chunked data from the server using PHP. For the 12mb file that's uploaded successfully without chunking, I'm handling it well using move_uploaded_file($_FILES['file']['tmp_name'], $url);. What code do I need to handle chunked data?

This is probably a max file size limitation on server side. Adjust server or php settings i think. — greenapps, Oct 24 '14 at 08:53
You should take a look at the server side error log. If PHP is configured properly, it should log its errors to the apache log file, check its config in the php.ini file, and the log file is probably somewhere under /var/log/apache or /var/log/httpd on the server side. — RealSkeptic, Oct 24 '14 at 09:01
@greenapps, phpinfo() indicates the upload_max_filesize is 512M so I don't think the upload size is a problem. — CL Loh, Oct 25 '14 at 15:15
@RealSkeptic, the above code does not produce any errors logged by server. I can confirm this as other errors are logged when the app sends requests to the PHP server. I'm guessing the request doesn't reach the server at all. — CL Loh, Oct 25 '14 at 16:36
If it doesn't reach the server at all, it wouldn't reach it for 12M either. Have you tried the new code with smaller file sizes? — RealSkeptic, Oct 25 '14 at 17:39
@RealSkeptic, smaller files e.g. jpg/png images are the same as the 12M video file - no errors when code to enable chunking is commented out `request.chunk(1024*1024)`, broken pipe error when chunking is enabled. — CL Loh, Oct 26 '14 at 16:15
On which device are you debugging your app? The old devices had a limit of 16MB of ram [link](http://stackoverflow.com/questions/18675557/what-is-the-maximum-amount-of-ram-an-app-can-use). — bardi, Jan 16 '15 at 16:47
Are you loading the entire request body in memory? w.r.t [HttpUrlConnection](http://developer.android.com/reference/java/net/HttpURLConnection.html) take into consideration that "When transferring large amounts of data to or from a server, use streams to limit how much data is in memory at once. Unless you need the entire body to be in memory at once, process it as a stream (rather than storing the complete body as a single byte array or string)" — bardi, Jan 16 '15 at 16:52
Hi @bardi. The file size doesn't seem to be a concern as much as I'd thought. I tried sending a small jpeg file with setChunkedStreamingMode enabled and the error occurs as well, though it worked when chunking is disabled. — CL Loh, Jan 19 '15 at 08:45

score 4 · Answer 1 · answered Jan 19 '15 at 11:54

As stated here you should call either setFixedLengthStreamingMode(len) when the body length is known in advance, or setChunkedStreamingMode(0) when it is not. About the broken pipe error, note that, if possible, android will reuse the old socket connection since establishing a connection is a resource consuming operation in a mobile environment. Try to force the POST to be performed over a new connection:

conn.setRequestProperty("connection", "close"); // disables Keep Alive

(note that the default behavior in android is to set that header to true and you can change it calling System.setProperty("http.keepAlive", "false");)

Guess this is the right answer, I was dealing with a similar problem. Using it was the solution. — MiguelHincapieC, Mar 19 '15 at 23:38
It took me 3 days to work this out, I wish I had found this post earlier. — Clive Jefferies, Mar 27 '15 at 16:58

Android to PHP: epipe Broken Pipe / stream closed posting chunked multipart form-data using httpurlconnection

1 Answers1