How is the reprojection error calculated in Matlab's triangulate function? Sadly, the documentation gives no mathematical formula

Question

How is the reprojection error calculated in Matlab's triangulate function?

Sadly, the documentation gives no mathematical formula.

It only says: The vector contains the average reprojection error for each M world point.

What is the procedure/Matlab uses when calculating this error?

I searched SOF but found nothing on this IMHO important question.

UPDATE: How can they use this error to filter out bad matches here : http://se.mathworks.com/help/vision/examples/sparse-3-d-reconstruction-from-two-views.html

Answer 1

AFAIK The reprojection error is calculated always in the same way (in the field of computer vision in general).

The reprojection is (as the name says) the error between the reprojected point in the camera and the original point.

So from 2 (or more) points in the camera you triangulate and get 3D points in the world system. Due to errors in the calibration of the cameras, the point will not be 100% accurate. What you do is take the result 3D point (P) and with the camera calibration parameters project it in the cameras again, obtaining new points (\hat{p}) near the original ones (p).

Then you calculate the euclidean distance between the original point and the "reprojected" one.

In case you want to know a bit more about the method used by Matlab, I'll enhance the bibliography they use giving you also the page number:

Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman (p312). Cambridge University Press, 2003.

But basically it is a least squares minimization, that has no geometrical interpretation.

Answer 2

Above mentioned answers interpret re-projection error in a simplistic way as an actual reprojection in the camera. In more general sense, this error reflects the distance between a noisy image point and the point estimated from the model. One can imagine a tangental plane to some surface (model) in n-dimensional space where the noisy point is projected (hence it lands on the plane, not on the model!). n is not obligatory = 2 since a notion of a "point" can be generalized to, for example, concatenation of coordinates of two corresponding points for Homography.

It is important to understand that reprojection error is not a final answer. Overall_error^2 = reprojection_error^2 + estimation_error^2. The latter is the distance between estimation reprojected and true point on the model. More on this can be found in chapter 5 of Hatrtley andd Zisserman book Multiple View Geometry. They prove that reprojeciton error has a theoretical limit 0.6*sigma (for Homography estimation), where sigma is noise standard deviation.

Answer 3

You can find an explanation of reprojection errors in the context of camera calibration in the Camera Calibrator tutorial:

The reprojection errors returned by the triangulate function are essentially the same concept.

The way to use the reprojection errors to discard bad matches is shown in this example:

[points3D, reprojErrors] = triangulate(matchedPoints1, matchedPoints2, ...
    cameraMatrix1, cameraMatrix2);

% Eliminate noisy points
validIdx = reprojErrors < 1;
points3D = points3D(validIdx, :);

This code excludes all 3D points for which the reprojection error was more than a pixel. You can also use validIdx to eliminate the corresponding 2D matches.

Answer 4

They filter out bad matches by removing relevant indexes which have large re-projection errors.

This means that points that have large re-projection errors are the outliers.