How to get Android camera preview data?

Question

My camera app displays a camera preview on the screen and also processes it in the background. Here is the relevant code, condensed as much as possible (e.g. no error handling or field declarations shown):

public final class CameraView extends SurfaceView implements
          SurfaceHolder.Callback, Runnable, PreviewCallback {

    public CameraView(Context context, AttributeSet attrs) {
        super(context, attrs);
        mHolder = getHolder();
        mHolder.addCallback(this);
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); 

    void openCamera() {
        // Called from parent activity after setting content view to CameraView
        mCamera = Camera.open();
        mCamera.setPreviewCallbackWithBuffer(this);
    }

    public void surfaceCreated(SurfaceHolder holder) {
        new Thread(this).start(); 

        // Set CameraView to the optimal camera preview size

        final Camera.Parameters params = mCamera.getParameters();
        final List<Camera.Size> sizes = params.getSupportedPreviewSizes();
        final int screenWidth = ((View) getParent()).getWidth();
        int minDiff = Integer.MAX_VALUE;
        Camera.Size bestSize = null;

        if (getResources().getConfiguration().orientation 
                == Configuration.ORIENTATION_LANDSCAPE) {
            // Find the camera preview width that best matches the
            // width of the surface.
            for (Camera.Size size : sizes) {
                final int diff = Math.abs(size.width - screenWidth);
                if (diff < minDiff) {
                    minDiff = diff;
                    bestSize = size;
                }
            }
        } else {
            // Find the camera preview HEIGHT that best matches the 
            // width of the surface, since the camera preview is rotated.
            mCamera.setDisplayOrientation(90);
            for (Camera.Size size : sizes) {
                final int diff = Math.abs(size.height - screenWidth);
                if (Math.abs(size.height - screenWidth) < minDiff) {
                    minDiff = diff;
                    bestSize = size;
                }
            }
        }

        final int previewWidth = bestSize.width;
        final int previewHeight = bestSize.height;

        ViewGroup.LayoutParams layoutParams = getLayoutParams();
        layoutParams.height = previewHeight;
        layoutParams.width = previewWidth;
        setLayoutParams(layoutParams);

        params.setPreviewFormat(ImageFormat.NV21);
        mCamera.setParameters(params);

        int size = previewWidth * previewHeight * 
            ImageFormat.getBitsPerPixel(params.getPreviewFormat()) / 8;
        mBuffer = new byte[size];
        mCamera.addCallbackBuffer(mBuffer);

        mCamera.setPreviewDisplay(mHolder);
        mCamera.startPreview();
    }

    public void onPreviewFrame(byte[] data, Camera camera) {
        CameraView.this.notify();
    }

    public void run() {
        mThreadRun = true;
        while (mThreadRun) {
            synchronized (this) {
                this.wait();
                processFrame(mBuffer); // convert to RGB and rotate - not shown
            }
            // Request a new frame from the camera by putting 
            // the buffer back into the queue
            mCamera.addCallbackBuffer(mBuffer);
        }

        mHolder.removeCallback(this);
        mCamera.stopPreview();
        mCamera.setPreviewCallback(null);
        mCamera.release();
        mCamera = null;
    }

    public void surfaceDestroyed(SurfaceHolder holder) {
        mThreadRun = false;
    }
}

On all devices, the camera preview displays properly, and on most (emulator, Samsung Galaxy S3, etc.) the data stored in mBuffer is also correct (after NV21 to RGB conversion and rotation, of course). However, a number of devices do not supply the correct data in onPreviewFrame. I'm sure that the data is being converted to RGB correctly after it's received, so the problem appears to be in the raw data supplied to mBuffer. I've noticed this bug report relating to the YV12 (alias YUV420p) camera preview format, but I'm using the old default, NV21 (alias YUV420sp), which must be supported according to the compatibility standard (see 7.5.3.2, bottom of page 29).

For example, for this scene (shown here in Camera Preview on the Samsung Galaxy Tab 2):

enter image description here

the data passed to mBuffer on the Tab 2 looks like:

enter image description here

and on the Motorola Droid 4 looks like:

enter image description here

What is the correct way to get Android camera preview data across all devices?

Edit: for processFrame(), I used OpenCV to convert to RGB and rotate. See this answer and this answer.

@EJTH I use OpenCV (method `cvtColor` with flag `COLOR_YUV420sp2RGBA`). — 1'', Jul 19 '13 at 02:11
So im guessing that you are executing the conversion to RGB on something that is not an android device? Out of curiosity does `YuvImage.compressToJpeg()` yield a similar distorted image? — EJTH, Jul 19 '13 at 02:19
@EJTH `YuvImage.compressToJpeg()` did work, and that fact led me to the correct answer. Put this into an answer and I'll award it the bounty. — 1'', Jul 19 '13 at 22:21

1'' · Accepted Answer · 2013-07-19T22:37:40.677

The only problem was that I didn't set the preview width and height:

params.setPreviewSize(previewWidth, previewHeight);
mCamera.setParameters(params);

This meant that the height and width I allocated for the array (proportional to previewWidth * previewHeight) tended to be a lot larger than the size of the actual data being returned (proportional to the default preview width and preview height). On some phones, the default was the same size as previewWidth and previewHeight, so there was no issue.

score 4 · Answer 2 · answered Feb 19 '14 at 11:37

You can also try this

public void takeSnapPhoto() {
camera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        Camera.Parameters parameters = camera.getParameters();
        int format = parameters.getPreviewFormat();
        //YUV formats require more conversion
        if (format == ImageFormat.NV21 || format == ImageFormat.YUY2 || format == ImageFormat.NV16) {
            int w = parameters.getPreviewSize().width;
            int h = parameters.getPreviewSize().height;
            // Get the YuV image
            YuvImage yuv_image = new YuvImage(data, format, w, h, null);
            // Convert YuV to Jpeg
            Rect rect = new Rect(0, 0, w, h);
            ByteArrayOutputStream output_stream = new ByteArrayOutputStream();
            yuv_image.compressToJpeg(rect, 100, output_stream);
            byte[] byt = output_stream.toByteArray();
            FileOutputStream outStream = null;
            try {
                // Write to SD Card
                File file = createFileInSDCard(FOLDER_PATH, "Image_"+System.currentTimeMillis()+".jpg");
                //Uri uriSavedImage = Uri.fromFile(file);
                outStream = new FileOutputStream(file);
                outStream.write(byt);
                outStream.close();
            } catch (FileNotFoundException e) {
                e.printStackTrace();
            } catch (IOException e) {
                e.printStackTrace();
            } finally {
            }
        }
    }
});}

This takes a picture and saves it to the SD card, which is different functionality than in the question. — 1'', Feb 19 '14 at 12:59
You save my time! Thanks! I needed to take two pictures very fast and this approach applicapable for me. It take me not more than 200 ms. between frames — Aleksey Timoshchenko, Dec 01 '16 at 13:34

score 2 · Answer 3 · answered Jul 18 '13 at 22:41

2

From working on Barcode Scanner, which relies heavily on preview data, I feel like I've seen every bug under the sun. My suggestion is simply to not call setPreviewFormat() and let it use the default. The default is what you want here fortunately. There seem to be fewer devices that fail to get the default right, than device that balls up the call to setPreviewFormat(). Try that at least, may or may not be it.

answered Jul 18 '13 at 22:41

Sean Owen

63,876
22
135
169

I've already tried that, and it has the same problem. Would you be interested in posting the relevant parts of your code? It might be possible to find the answer by comparing the two methods. – 1'' Jul 18 '13 at 22:54
It is all on http://code.google.com/p/zxing - look for CameraConfigurationManager. – Sean Owen Jul 18 '13 at 23:36
Thanks. Your code uses `setOneShotPreviewCallback` instead of `setPreviewCallbackWithBuffer`, so I tried switching, and the images look the exact same. I'm increasingly wondering whether there's simply a gigantic bug in the way Android integrates with these devices' cameras. – 1'' Jul 19 '13 at 01:45
Does Barcode Scanner work on these? At least that would barrow it down. If it does maybe the driver is not hopeless even if there is some bug. – Sean Owen Jul 19 '13 at 12:28

How to get Android camera preview data?

3 Answers3

Linked