Android camera preview is always sideways

Question

I am writing an Android plugin that will take data from the camera preview and send it to Unity. (For various reasons, I am not able to use the Unity WebCamTexture) I am able to get the camera preview data and sent it to Unity, however, the image is always rotated 90 degrees. My Unity app is set to always be in portrait mode.

On my Pixel XL, the front and back images are rotated in opposite directions. Here is a photo of my app when using the front and back facing cameras.

I have created the function AdjustOrientation in NativeCamera.java in an attempt to fix the orientation, but calling it does not seem to to have any effect.

I have found links to places that say to add in code similar to what is in the AdjustOrientation function to fix the problem, but none of them have solved the problem.

Here is a link I have investigated that did not solve my problem.

Android - Camera preview is sideways

I have tried different variations of

mCamera.setDisplayOrientation()
params.setRotation()

but have not had any luck.

I only need the image to be in the correct orientation in

public void onPreviewFrame(byte[] data, Camera camera)

It does not matter if a saved image would be upside down or rotated. I am simply passing the data on to my Unity Project and the only purpose of the Android Plugin is to pass the camera data to Unity. As long as the image data is correct inside OnPreviewFrame I would be all set.

I know there is a newer Camera2 API and that it deprecates the original Camera, but I would really like to be able to fix this with my existing plugin without having to write a new plugin that uses Camera2.

Here is a link to my project.

https://drive.google.com/open?id=1MD-NRVf0YhhVIiRUOiBptvSwh9wtK3V7

Thanks in advance.

John Lawrie

Updates. Here is the source of my .java file for easier reference. If you look in AdjustOrientation you can see I have some code to try to adjust displayorientation. It appears to have no effect.

package com.test.camerapreview;

import android.annotation.SuppressLint;
import android.app.Activity;
import android.graphics.SurfaceTexture;
import android.hardware.Camera;
import android.opengl.GLES11Ext;
import android.opengl.GLES20;
import android.support.annotation.Dimension;
import android.util.Log;
import android.util.Size;
import android.view.Display;
import android.view.Surface;

import com.unity3d.player.UnityPlayer;
import com.google.gson.Gson;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class NativeCamera implements Camera.PreviewCallback {

    public static NativeCamera instance;
    public static String gameObjectTargetName;
    private static Activity myActivity;

    Camera mCamera;
    SurfaceTexture texture;
    int nativeTexturePointer = -1;
    int prevHeight;
    int prevWidth;

    //
    // Call this function first.
    //
    public static void Setup(String gameObjectName, Activity theActivity){
        gameObjectTargetName = gameObjectName;
        myActivity = theActivity;
        instance = new NativeCamera();
    }

    public int startCamera(int idx, int width, int height) {
        nativeTexturePointer = createExternalTexture();
        texture = new SurfaceTexture(nativeTexturePointer);

        mCamera = Camera.open(idx);

        setupCamera(idx, width, height);

        try {
            mCamera.setPreviewTexture(texture);
            mCamera.setPreviewCallback(this);

            AdjustOrientation(width, height);

            mCamera.startPreview();
            Log.i("Unity", "JAVA: camera started");
        } catch (IOException ioe) {
            Log.w("Unity", "JAVA: CAM LAUNCH FAILED");
        }
        return nativeTexturePointer;
    }

    public void stopCamera() {
        mCamera.setPreviewCallback(null);
        mCamera.stopPreview();
        mCamera.release();
        GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
        Log.i("Unity", "JAVA: Camera stopped");
    }

    private int createExternalTexture() {
        int[] textureIdContainer = new int[1];
        GLES20.glGenTextures(1, textureIdContainer, 0);
        GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textureIdContainer[0]);
        return textureIdContainer[0];
    }

    @SuppressLint("NewApi")
    private void setupCamera(int cameraID, int width, int height) {
        Camera.Parameters params = mCamera.getParameters();

        params.setRecordingHint(true);
//      params.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
        params.setPreviewFormat(17);
        params.setZoom(0);
            // 16 ~ NV16 ~ YCbCr
            // 17 ~ NV21 ~ YCbCr ~ DEFAULT *
            // 4  ~ RGB_565
            // 256~ JPEG
            // 20 ~ YUY2 ~ YcbCr ...
            // 842094169 ~ YV12 ~ 4:2:0 YCrCb comprised of WXH Y plane, W/2xH/2 Cr & Cb. see documentation *
//      params.setFlashMode(Camera.Parameters.FLASH_MODE_TORCH);
        Camera.Size previewSize = getOptimalSize(width, height, mCamera.getParameters().getSupportedPreviewSizes());
        int previewWidth = previewSize.width;
        int previewHeight = previewSize.height;
        params.setPictureSize(previewWidth, previewHeight);
        params.setPreviewSize(previewWidth, previewHeight);
        params.setWhiteBalance(Camera.Parameters.WHITE_BALANCE_AUTO);
        params.setExposureCompensation(0);

        //
        // Fix the orientation.
        //
//        int orientation = detectCameraDisplayOrientation(cameraID);
//        mCamera.setDisplayOrientation(orientation);
//        params.setRotation(orientation);

//        Camera.CameraInfo info = new Camera.CameraInfo();
//        Camera.getCameraInfo(cameraID, info);

//        Log.d("Unity", "info.orientation = " + info.orientation);

//        params.setRotation(-90);
//        mCamera.setDisplayOrientation(-90);

//        mCamera.setDisplayOrientation(90);
//        params.setRotation(90);


        try{
            mCamera.setParameters(params);
        } catch (Exception e){
            Log.i("Unity", "ERROR: " + e.getMessage());
        }

        Camera.Size mCameraPreviewSize = params.getPreviewSize();
        prevWidth = mCameraPreviewSize.width;
        prevHeight = mCameraPreviewSize.height;

        int[] fpsRange = new int[2];
        params.getPreviewFpsRange(fpsRange);
        String previewFacts = mCameraPreviewSize.width + "x" + mCameraPreviewSize.height;
        if (fpsRange[0] == fpsRange[1]) {
            previewFacts += " @" + (fpsRange[0] / 1000.0) + "fps";
        } else {
            previewFacts += " @[" + (fpsRange[0] / 1000.0) + " - "  + (fpsRange[1] / 1000.0) + "] fps";
        }
        Log.i("Unity", "JAVA: previewFacts=" + previewFacts);
    }

    private void AdjustOrientation(int width, int height) {
        Camera.Parameters parameters = mCamera.getParameters();

        Display display = myActivity.getWindowManager().getDefaultDisplay();

        if(display.getRotation() == Surface.ROTATION_0) {
            Camera.Size previewSize = getOptimalSize(height, width, mCamera.getParameters().getSupportedPreviewSizes());
            prevWidth = previewSize.width;
            prevHeight = previewSize.height;

            mCamera.setDisplayOrientation(90);
        }

        else if(display.getRotation() == Surface.ROTATION_90) {
            Camera.Size previewSize = getOptimalSize(width, height, mCamera.getParameters().getSupportedPreviewSizes());
            prevWidth = previewSize.width;
            prevHeight = previewSize.height;
        }

        else if(display.getRotation() == Surface.ROTATION_180) {
            Camera.Size previewSize = getOptimalSize(height, width, mCamera.getParameters().getSupportedPreviewSizes());
            prevWidth = previewSize.width;
            prevHeight = previewSize.height;
        }

        else { //if(display.getRotation() == Surface.ROTATION_270) {
            Camera.Size previewSize = getOptimalSize(width, height, mCamera.getParameters().getSupportedPreviewSizes());
            prevWidth = previewSize.width;
            prevHeight = previewSize.height;

            mCamera.setDisplayOrientation(180);
        }

        parameters.setPreviewSize(prevWidth, prevHeight);

        mCamera.setParameters(parameters);
    }

    private Camera.Size getOptimalSize(int width, int height, List<Camera.Size> sizes) {
        if(mCamera == null)
            return null;

        final double ASPECT_TOLERANCE = 0.1;
        double targetRatio=(double)width / height;

        if (sizes == null)
            return null;

        Camera.Size optimalSize = null;
        double minDiff = Double.MAX_VALUE;
        int targetWidth = width;

        for (Camera.Size size : sizes) {
            double ratio = (double) size.width / size.height;
            Log.i("Unity", "RES: size=" + size.width + "/" + size.height + " Aspect Ratio: " + ratio + " target width: " + width + " target height: " + height);

            if (Math.abs(ratio - targetRatio) > ASPECT_TOLERANCE) {
                continue;
            }

            if (Math.abs(size.width - targetWidth) < minDiff) {
                optimalSize = size;
                minDiff = Math.abs(size.width - targetWidth);
            }
        }

        if (optimalSize == null) {
            minDiff = Double.MAX_VALUE;
            for (Camera.Size size : sizes) {
                if (Math.abs(size.width - targetWidth) < minDiff) {
                    optimalSize = size;
                    minDiff = Math.abs(size.width - targetWidth);
                }
            }
        }
        Log.i("Unity", "optimal size=" + optimalSize.width + "/" + optimalSize.height + "/ Aspect Ratio: " + (double) optimalSize.width / optimalSize.height);
        return optimalSize;
    }

    public int getPreviewSizeWidth() {
        return prevWidth;
    }

    public int getPreviewSizeHeight() {
        return prevHeight;
    }

    public String GetPreviewSizes(int id) {
        Camera cam = Camera.open(id);
        Camera.Parameters params = cam.getParameters();

        Gson gson = new Gson();
        String JSON = gson.toJson(params.getSupportedPreviewSizes());

        cam.release();

        Log.d("Unity", "Supported sizes are " + JSON);

        return JSON;
    }

    public byte[] bytes;
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        bytes = data;

        UnityPlayer.UnitySendMessage(gameObjectTargetName, "GetBuffer", "");
    }

}

have look [this](https://stackoverflow.com/a/28528522/5110595) — Hemant Parmar, Jan 01 '18 at 04:48
In that link, I see that it mentions 2 things. The first is to make sure that portrait mode is specified in the manifest. It is. The other thing is some code to adjust the display orientation. My AdjustOrientation code does essentially that already. I'm going to edit my questions to include the code. — Dalanchoo, Jan 01 '18 at 05:55
Unfortunately, both old and new Camera API deliver the preview pixels in their natural order. You can rotate the frame received at **onPreviewFrame()** manually, but this can be really slow. It's much better to 'teach' your Unity component to rotate the image for display. — Alex Cohn, Jan 01 '18 at 13:34

score 1 · Answer 1 · answered Jan 01 '18 at 22:53

Despite spending a lot of time trying to get the camera to rotate the image with SetDisplayOrientation, for some reason none of that had any effect. I ended up doing what Alex Cohen suggested and put manually rotated the image in OnPreviewFrame

This is a link to the page that gives the code to how to rotate the image.

Rotate an YUV byte array on Android

Santanu Sur · Answer 2 · 2018-01-01T06:45:09.993

0

in camera preview... set `mCamera.setPreviewDisplay(holder);

        mCamera = Camera.open(0);
        mCamera.setPreviewDisplay(holder);
        mCamera.startPreview();
        mCamera.setDisplayOrientation(90);`

edited Jan 01 '18 at 06:45

answered Jan 01 '18 at 04:22

Santanu Sur

8,836
7
24
43

Are you talking about the function OnPreviewFrame()? At that point, we already have the data passed in as a parameter so this seems too late. Also, there is no mCamera.setRotation() function. There is a setDisplayOrientation and a cameraParams.setRotation() Is that what you are referring to? – Dalanchoo Jan 01 '18 at 06:10
in the cameraPreview class ... `params.setRotation(90);` my mistake – Santanu Sur Jan 01 '18 at 06:12
i am not sitting in front of pc right now... just check where the setrotation func is i guess its in the cameraPreview class just pass `90` as param.. – Santanu Sur Jan 01 '18 at 06:15
i have updated my answer `mCamera.setDisplayOrientation(90)` do that it will work.. – Santanu Sur Jan 01 '18 at 06:45
and dude `mCamera.open(1)` for using front camera. – Santanu Sur Jan 01 '18 at 06:47
use that `onSurfaceCreated()` – Santanu Sur Jan 01 '18 at 06:55
you can do one more thing ... log and check that if at all `if(display.getRotation==Surface.Rotation_0)` is true .. run it in debug mode.. – Santanu Sur Jan 01 '18 at 07:08
Per the API reference, setDisplayOrientation() does nothing for onPreviewFrame. – Eddy Talvala Jan 02 '18 at 20:38

score 0 · Answer 3 · answered Jan 02 '18 at 20:37

Camera.setDisplayOrientation only affects the preview output that's passed to Camera.setPreviewDisplay or Camera.setPreviewTexture. That's documented in the API reference for setDisplayOrientation:

This does not affect the order of byte array passed in onPreviewFrame(byte[], Camera), JPEG pictures, or recorded videos.

If you need to adjust the output of onPreviewFrame, then you need to do it yourself. The rotation you need to apply is the same value you'd pass to setDisplayOrientation, in the clockwise direction.

Android camera preview is always sideways

3 Answers3