Unreal simulated data (CARLA) to generate depth map using OpenCV

Question

I'm working with a couple of RGB cameras (+depth cameras) in CARLA simulator (Unreal Engine environment). The cameras are attached to a vehicle (the position is relative to the car) at [0.5,-1,1.8] and [0.5, 1, 1.8] (left and right respectively, Unreal coordinate system). While the first camera is considered fixed during the simulation, the second camera position is perturbed by Gaussian noise in both position and rotation.

The cameras are horizontally displaced like a common stereo setup. Intuitively, the relative position of the second camera is [0+n_x, 2+n_y, 0+n_z] where n_x, n_y, n_z is the Gaussian noise in the three direction. Changing the coordinates to the OpenCV ones, the resulting relative position is [2+n_y, -n_z, n_x] (the minus sign on n_z come from the fact that the axis are inverted). Basically, I run the following code to get the relative position in OpenCV:

    relative_pos[0] = -camL_pos.at<double>(1) + camR_pos.at<double>(1); //baseline ~2mt
    relative_pos[1] = -camL_pos.at<double>(2) + camR_pos.at<double>(2); //height difference
    relative_pos[2] =  camL_pos.at<double>(0) - camR_pos.at<double>(0);

Since OpenCV expect a translation vector and not the coordinates with respect to the first camera, this vector is multiplied by -1. Other than the position, also a rotation perturbation is applied in the three angles, roll, pitch and yaw by mean of gaussian noise computed separately. The values of the noise for the position and rotation are stored for every frame.

What I would like to compute, is the disparity map using OpenCV. I know that I have a depth camera, but I use it just as a ground truth.

After all this introduction, here are the questions:

Is the formulation of the translation above correct, or am I missing something swtiching from a coordinate system to another?
How should I manage the rotation? I'm really confused here. I haven't understood if I need to transform the angles or not (I suppose that at least the yaw value should be inverted due the opposite direction of the two axis in the two coordinate system).
What should I pass to the stereoRectify function? The relative position defined by t and the relative rotation R are fine?

Thank you everyone in advance.

EDIT (additional information):

Unreal's transform order: Scale -> Rotation -> Translation

Regarding the rotation, the order is roll->pitch->yaw

Unreal simulated data (CARLA) to generate depth map using OpenCV

0 Answers0