6

For recording audio and video, I am creating webm files under the ondataavailable of MediaRecorder API. I have to play each created webm file individually.

Mediarecorder api inserts header information into first chunk (webm file) only, so rest of the chunks do not play individually without the header information.

As suggested link 1 and link 2, I have extracted the header information from first chunk,

// for the most regular webm files, the header information exists
// between 0 to 189 Uint8 array elements

const headerIinformation = arrayBufferFirstChunk.slice(0, 189); 

and perpended this header information into second chunk, still the second chunk could not play, but this time the browser is showing poster (single frame) of video and duration of sum of two chunks, eg:10 seconds; duration of each chunk is 5 second.

The same header-information thing I have done with the hex editor. I opened the webm file in editor and copied the first 190 elements from first webm file and put this into second file, something like below image, even this time, the second webm file could not play and the result was same as in previous example.

Red color is showing the header information:

webm hex

This time I copied the header and cluster information from first webm file placed this into second file, something like below image, but did not get success,

webm hex

Questions

What I am doing wrong here ?

Is there any way that we can play the webm files/chunks individually ?

Note: I can't use the MediaSource to play those chunks.

Edit 1

As @Brad suggested, I want to insert all the content before the first cluster to a later a cluster. I have few webm files that each has duration of 5 seconds. After digging into the files, I came to know, almost every alternate file hasn't cluster point (no 0x1F43B675).

Here I am confused that I'll have to insert header information (initialization data) at the beginning of every file or beginning of every first cluster? If I choose a later option, then how's going to play the webm file that doesn't have any cluster ?

Or, first I need to make each webm file in a way that it has cluster at very beginning, so I can prepend the header information before cluster in those files?

Edit 2

After some digging and reading this , I came up with the conculsion that each webm file needs header info, cluster and actual data.

Suman Bogati
  • 5,931
  • 1
  • 18
  • 33
  • Why do you have to play these chunks individually? Why can't you simply create a new MediaRecorder per "chunk"? You could still have the main one running in parallel to record the whole as a single file, or even just use server-side post processing to concat all these files as a single one. – Kaiido Jun 07 '20 at 01:21

2 Answers2

4

// for the most regular webm files, the header information exists

// between 0 to 189 Uint8 array elements

Without seeing the actual file data it's hard to say, but this is possibly wrong. The "header information" needs to be everything up to the first Cluster element. That is, you want to keep all data from the start of the file up to before you see 0x1F43B675 and treat it as initialization data. This can/will vary from file to file. In my test file, this occurs a little after 1 KB in.

and perpended this header information into second chunk, still the second chunk could not play, but this time the browser is showing poster (single frame) of video and duration of sum of two chunks, eg:10 seconds; duration of each chunk is 5 second.

The chunks output from the MediaRecorder aren't relevant for segmentation, and can occur at various times. You would actually want to split on the Cluster element. That means you need to parse this WebM file, at least to the point of splitting out Clusters when their identifier 0x1F43B675 comes by.

Is there any way that we can play the webm files/chunks individually ?

You're on the right path, just prepend everything before the first Cluster to a later Cluster.

Once you've got that working, the next problem you'll likely hit is that you won't be able to do this with just any cluster. The first Cluster must begin with a keyframe or the browser won't decode it. Chrome will skip over to the next cluster, to a point, but it isn't reliable. Unfortunately, there's no way to configure keyframe placement with MediaRecorder. If you're lucky enough to be able to process this video server-side, here's how to do it with FFmpeg: https://stackoverflow.com/a/45172617/362536

Community
  • 1
  • 1
Brad
  • 146,404
  • 44
  • 300
  • 476
  • 1
    Hi Brad, does it mean to be able to real-time stream WEBM you still need server-side processing no pure-client side approach? – quarks Jun 06 '20 at 21:51
  • 1
    @quarks You could re-mux client-side. I don't have any specific code for you for that... I started down that road at one point but ended up going server-side as I needed to transcode for other bitrates anyway. You could also figure out a way to determine what Cluster elements start with keyframes, and only allow seeking to them. Maybe this is as simple as finding large clusters. A keyframe has to start at the beginning of a Cluster, but not all Clusters start with a keyframe. – Brad Jun 06 '20 at 22:49
  • Are you actually using that approach in production? What is your transport, TCP or UDP? – quarks Jun 07 '20 at 01:30
  • @Brad, I really appreciate your reply. I don't have the privilege of video processing on the server side. I have webm files with each having a duration of 5 seconds. After digging into the files, I came to know, almost every alternate file hasn't cluster point (no 0x1F43B675) Here I am confused that I'll have to insert at the beginning of every file or beginning of every first cluster? If I choose a later option, then how's going to play the webm file which doesn't have any cluster ? – Suman Bogati Jun 07 '20 at 08:23
  • @Brad, as I totally stuck, can you please suggest about the updated question (Edit 1) ? – Suman Bogati Jun 08 '20 at 13:54
  • @Brad, would you like to see my new question https://stackoverflow.com/questions/62302652/webm-file-could-not-play-when-mediarecorder-on-chrome-does-not-provide-the-clust ? – Suman Bogati Jun 11 '20 at 04:38
0

Okay looks like this is not as easy as you have to scan through the blob to find the magic value.

let offset = -1;
let value = 0;
const magicNumber = parseInt("0x1F43B675".match(/[a-fA-F0-9]{2}/g).reverse().join(''), 16)

while(value !== magicNumber) {
  offset = offset + 1;

  try {
    const arr = await firstChunk.slice(offset, offset + 4).arrayBuffer().then(buffer => new Int32Array(buffer));
    value = arr[0];
  }
  catch(error) {
    return;
  }
}

offset = offset + 4;

The answer is 193 199

const header = firstChunk.slice(0, offset);
const blobType = firstChunk.type;
const blob = new Blob([header, chunk], { type: blobType });

And there you have it. Now question is how did I get this number? Why is it not multiple of 42?

Brute force

Well the logic is simple, record the video, gather chunks, slice the first chunk, compute new blob and try to play it with HTMLVideoElement. If it fails increase the offset.

(async() => {

    const microphoneAudioStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true });

    const mediaRecorder = new MediaRecorder(microphoneAudioStream);
    let chunks = [];

    mediaRecorder.addEventListener('dataavailable', (event) => {
        const blob = event.data;
        chunks = [...chunks, blob];
    });

    mediaRecorder.addEventListener("stop", async () => {
        const [firstChunk, ...restofChunks] = chunks;
        const [secondBlob] = restofChunks;
        const blobType = firstChunk.type;

        let index = 0;
        const video = document.createElement("video");

        while(index < 1000) {
            const header = firstChunk.slice(0, index);
            const blob = new Blob([header, secondBlob], { type: blobType });
            const url = window.URL.createObjectURL(blob);

            try {
                video.setAttribute("src", url);
                await video.play();
                console.log(index);
                break;
            }
            catch(error) {

            }

            window.URL.revokeObjectURL(url);

            index++;
        }
    })

    mediaRecorder.start(200);

    const stop = () => {
        mediaRecorder.stop();
    }

    setTimeout(stop, 400)

})();

I noticed that for smaller timeslice param in MediaRecorder.start and timeout param in setTimeout the header offset becomes 1. Sadly still not 42.

Józef Podlecki
  • 6,858
  • 3
  • 15
  • 33