11

I am trying to get each frame from a TextureView, unfortunately trying:

textureView.getBitmap();

Results in slow performance is there a faster way to obtain a bitmap. Is it better to use the NDK instead?

Looking for actual examples

fadden
  • 48,613
  • 5
  • 104
  • 152
AndyRoid
  • 4,754
  • 8
  • 33
  • 70

1 Answers1

4

A TextureView receives frames on a SurfaceTexture, which takes frames sent to its Surface and converts them to a GLES texture. To get the pixel data out, the texture must be rendered to a framebuffer, then read out with glReadPixels(). The pixel data can then be wrapped with a Bitmap object (which may or may not involve copying the pixel data).

Using the NDK isn't going to do you much good, as all of the code that needs to run quickly is already implemented natively.

You may see some improvement by sending the data directly to a SurfaceTexture and doing the GLES work yourself, but presumably you want to display the incoming frames in the TextureView, so all you'd potentially save is the Bitmap overhead (which may or may not be significant).

It might help if you explained in your question where the frames are coming from and what it is you want to do with them.

fadden
  • 48,613
  • 5
  • 104
  • 152
  • I want to do some processing on the frame's think image recognition. So real time recognition on the frames the issue right now is that calling .getBitmap() takes roughly ~50-60 ms. I can solve this by making the bitmap smaller which reduces the time but then I also want to redisplay the frames back into the texture view. How would you propose I go about doing the GLES work myself? – AndyRoid Mar 28 '16 at 22:39
  • GLES rendering + `glReadPixels()` should be much faster. If your display requirements are simple (on top of the View layer or entirely below it) you can just render the texture to a SurfaceView. So you'd direct your incoming frames to a SurfaceTexture and render the texture to a SurfaceView, calling `glReadPixels()` before `eglSwapBuffers()` to grab a copy of the pixels. That'll give you a direct byte buffer with RGB pixels, which you can use from Java code or C++. I don't have an exact example to offer, but the various pieces can be found in Grafika (https://github.com/google/grafika). – fadden Mar 28 '16 at 23:09
  • Thank you for the detailed answer! – AndyRoid Mar 28 '16 at 23:47
  • If you can directly link me to an example that would be sweet. – AndyRoid Mar 28 '16 at 23:47
  • Also you mentioned `AsyncTask` can have a direct hit on performance due to lower priority I noticed that in the Grafika `ReadPixelsActivity` they are using an Asynchronous Task, would a `Handler` be a better alternative? – AndyRoid Mar 28 '16 at 23:50
  • I found out about AsyncTask after the code was written; note the comment here: https://github.com/google/grafika/blob/master/src/com/android/grafika/ReadPixelsActivity.java#L138 . You can set up a SurfaceView renderer thread and do all of the work there. All of the Grafika examples use Looper/Handler to manage interactions between the main thread and the worker thread. – fadden Mar 28 '16 at 23:59
  • Awesome thanks, also I noticed that there is a `Matrix` class inside OpenGL can I also use that for computation intensive work? Possible on every frame? – AndyRoid Mar 29 '16 at 00:21
  • If you want to use the GPU for computation you'll want to look into RenderScript. The Matrix class is just a utility class for creating and modifying matrices to hand to GLES. – fadden Mar 29 '16 at 02:20
  • Any good examples with `RenderScript` and OpenGL? I want to render video with OpenGL and do computations with `RenderScript`. I've used `RenderScript` for basic image manipulation before hand but not computations – AndyRoid Mar 29 '16 at 02:26
  • Sorry, I don't have any examples to recommend. – fadden Mar 29 '16 at 15:46
  • 1
    Hey @fadden coming back to this today, looking for a specific example of what you are talking about in regards to extracting frames from a `SurfaceTexture`. Specifically I ported `GLSurfaceView` to a `GLTextureView`, how would I go about extracting frames from a MP4 set into a custom `GLTextureView` that plays videos? How would I even get the video to render to the `GLTextureView` appropriately? – AndyRoid Apr 05 '16 at 22:38