How to use MediaCodec without MediaExtractor for H264

Question

I need to use MediaCodec without the MediaExtractor and I'm reading the file using a FileInputStream. Currently it is not working, it is showing a greenish scrambled image on the screen.

This is the whole source code:

FileInputStream in = new FileInputStream("/sdcard/sample.ts");

String mimeType = "video/avc";
MediaCodec decoder = MediaCodec.createDecoderByType(mimeType);
MediaFormat format = MediaFormat.createVideoFormat(mimeType, 1920, 1080);

byte[] header_sps = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108 };
byte[] header_pps = { 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(header_sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(header_pps));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

decoder.configure(format, surface, null, 0);
decoder.start();

ByteBuffer[] inputBuffers = decoder.getInputBuffers();
ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
BufferInfo info = new BufferInfo();
boolean isEOS = false;
long startMs = System.currentTimeMillis();

while (!Thread.interrupted()) {
    if (!isEOS) {
        int inIndex = decoder.dequeueInputBuffer(1000);
        if (inIndex >= 0) {
            byte buffer2[] = new byte[18800 * 8 * 8 * 8];
            ByteBuffer buffer = inputBuffers[inIndex];
            int sampleSize;

            sampleSize = in.read(buffer2, 0, 18800 * 4);

            buffer.clear();
            buffer.put(buffer2, 0, sampleSize);
            buffer.clear();

            if (sampleSize < 0) {
                decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                isEOS = true;
            } else {
                decoder.queueInputBuffer(inIndex, 0, sampleSize, 0, 0);
            }
        }
    }

    int outIndex = decoder.dequeueOutputBuffer(info, 10000);
    switch (outIndex) {
    case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
        Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
        outputBuffers = decoder.getOutputBuffers();
        break;
    case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
        Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());
        break;
    case MediaCodec.INFO_TRY_AGAIN_LATER:
        Log.d("DecodeActivity", "dequeueOutputBuffer timed out! " + info);
        break;
    default:
        ByteBuffer buffer = outputBuffers[outIndex];
        Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);

        while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
            try {
                sleep(10);
            } catch (InterruptedException e) {
                e.printStackTrace();
                break;
            }
        }
        decoder.releaseOutputBuffer(outIndex, true);
        break;
    }

    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
        Log.d("DecodeActivity", "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
        break;
    }
}

decoder.stop();
decoder.release();

If I use the MediaExtractor, everything works fine. I got the SPS/PPS values by looking at the MediaFormat when using MediaExtractor. If I remove the section below, nothing is shown on the screen.

byte[] header_sps = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108 };
byte[] header_pps = { 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(header_sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(header_pps));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

What I am missing? How can I get the SPS/PPS values programatically without MediaExtractor?

Are SPS and PPS values you used standard values? In my case, I am getting the data in the form of NAL units, so i am searching for frames with NAL type 7 and 8 for PPS and SPS. Instead can I use the above values directly? Also, I did not understand the posted answer clearly. Can you explain what exactly was the problem with your code? I am using this code as a reference for my scenario. — mk.., Sep 01 '16 at 11:14
No, each stream has its own SPS/PPS values, you should get them like you mentioned above. The problem was that I was sending to MediaCodec (queueInputBuffer) a fixed-size buffer (with size 18800 * 4), but instead you should send complete video frames only. Is it clear now? — thiagolr, Sep 02 '16 at 10:32
There is a way to exclude `MediaExtractor` and put `byte[]` frame (one by one) directly to the inputBuffer (`ByteBuffer`). Is this what you actually want? — Volodymyr Kulyk, Apr 27 '17 at 07:17

score 11 · Accepted Answer · answered Nov 02 '13 at 14:45

11

I'm assuming you're reading a raw H.264 elementary stream and not an MP4 file.

It looks like you're feeding fixed-size blocks of data to the decoder. That doesn't work. You need to put a single access unit into each buffer.

answered Nov 02 '13 at 14:45

fadden

48,613
5
104
152

That was the problem! Now I'm feeding the decoder only with video packets and it is working! – thiagolr Nov 06 '13 at 09:11
3

@thiagolr Can you post the solution? How you solved this.. I am stuck at `decoder.dequeueOutputBuffer(info, 10000);` I always get -1... – Patrick Leijser Apr 14 '17 at 13:35

Ganesh · Answer 2 · 2013-11-03T06:17:16.520

6

To your last question i.e. how can you get SPS and PPS values, you need to have a mechanism to read the same from the file.

If you are having an elementary stream file, then you would need to parse the file, identify NALU headers and extract the content.

If you have container format, you will need to have a mechanism to read the file format of the container type and extract the information.

If you have a streaming input, then you can receive the content from the incoming SDP information.

As for your current code, I would recommend concatenating both SPS and PPS into one buffer and provide the same to the underlying codec as shown below

byte[] csd_info = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108, 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(csd_info));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

edited Nov 03 '13 at 06:17

answered Nov 02 '13 at 14:47

Ganesh

5,790
2
34
54

1

The main issue was fixed with the other (accepted) answer. But thanks for answering the second question! +1 for that! – thiagolr Nov 06 '13 at 09:13
1

MediaCodec codecs appear to differ among devices about what constitutes a NAL unit. Some want the 0001 pattern at the beginning, others not. Others are fussy about what you do with non-VCL units. – Paul Steckler Feb 28 '15 at 02:19
@PaulSteckler.. It's the implementation of the decoders which is making a difference. Some codecs stick to `RTP` standard and don't handle a start code. An example of this is `H.264` streaming to `Quicktime` which doesn't display the video if a start code is present as `RTP` standard mandates that first byte is the `NALU`type. Ideally, a decoder should be capable of searching for a start code and be robust enough to handle variations. – Ganesh Mar 02 '15 at 04:28

How to use MediaCodec without MediaExtractor for H264

2 Answers2

Linked