5

I am trying to compute the coordinates correspondence of several points between two images. I have a group of points whose correspondences are known, I use them with OpenCV's findFundamentalMatrix() in order to find the fundamental matrix. I verified that x^T * F * x' = (0) for each point, and the result is always right or very close.

The thing is, now I'd like to use the coordinates of a point on the first image (y) and the fundamental matrix (F) in order to find the coordinates of the point on the second image (y'). I first thought about simply using the equation above, but given only the z of the y' point, there can be an infinity of solutions.

How else can I use the fundamental matrix to compute the translations ?

To be more clear: knowing the fundamental matrix "linking" two projections, how can I use it to translate the coordinates of any known point (a, b, 1) from the first projection to the second projection?

Considering that we know a, b and F in this equation: (a', b", 1)T * F * (a, b, 1) = (0)

I had made a simple drawing as an example: http://i.imgur.com/drNr2.jpg . The idea is to find the coordinates of the red dot (xq, yq) in projection 2, considering that we know its coordinates in projection 1 and the ones of all other points in both projections (and some other ones as the algorithm to find the fundamental matrix actually requires at least 8 points)

Another precision: in my example, known points are coplanar, but the researched point will not necessarily be.

I hope that made my problem more clear :)

Community
  • 1
  • 1
Jukurrpa
  • 3,798
  • 6
  • 40
  • 70

3 Answers3

3

The fundamental matrix transforms points from one image to lines in the other. Could you elaborate more on

How else can I use the fundamental matrix to compute the translations?

please. Telling us what you want to achieve perhaps with an example would help too.

Edit: If you have calibrated the camera you can compute the essential matrix, E, from the fundamental matrix, F. E transforms a point in one image to a point in the other. But of course, the requirement is to have the internal matrix. If K is the internal matrix E=transpose(K)FK. The other method is to find the corresponding line for a point in the other image and then search along this line for the patch most similar in appearance to the patch surrounding the point in the first image. There are some other ways too but really need more information about the problem to tell which suits your case.

Edit 2: in the drawing you have got the points are coplanar. Hence, a homography maps the point positions between the two images, and there is no need to find the fundamental matrix. OpenCV has a function for estimating homographies, which needs only four points.

fireant
  • 12,032
  • 4
  • 34
  • 47
  • if the group of points you used are from a chessboard, you can easily use opencv to get the internal parameters (it requires a few poses of the chessboard as far as I remember) – guinny Apr 30 '12 at 17:07
  • This looks more like a comment than an answer :) – Adiel Mittmann Apr 30 '12 at 17:08
  • :-) that got too long so decided to make it an answer. – fireant Apr 30 '12 at 17:14
  • I edited my question to add some information, I hope it will help. I haven't tried OpenCV's calibration, does it require any manual input regarding the position of the two cameras ? – Jukurrpa Apr 30 '12 at 18:13
  • Please see my second edit. In this case, points are always coplanar? – fireant Apr 30 '12 at 18:24
  • The known points are coplanar, but the one I'm searching for will not necessarily be. (will never be, actually) – Jukurrpa May 02 '12 at 08:58
  • So, do you have any ideas on how I could do that ? – Jukurrpa May 03 '12 at 09:32
  • To recap, all the point with known correspondences are coplanar (all on a single plane) and the points you want to map from the first image to the second are not on that plane, right? Is that all you have got? for example you don't have inaccurate/putative coresponses for the points you want to map? And finally you cant calibrate the camera? – fireant May 03 '12 at 12:59
  • the reason for asking all those is that if all known points/correspondences are coplanar and the rest of points are not and the camera is uncalibrated, and the camera is projective (not affine) then I cant think of a way to project the points from the first image to the other. – fireant May 03 '12 at 13:03
  • Yes, that is all I got. What do you mean by putative corresponses ? As for the calibration I just haven't looked into it (and have a hard time understanding its purpose) can it be used efficiently when the cameras are physically far away from each other? Also, what is the difference between an affine and a projective camera? (as you might have guessed my knowledge in this domain isn't great for now) – Jukurrpa May 03 '12 at 14:50
  • I cannot edit my comment anymore, but I got it for affine and projective cameras. – Jukurrpa May 04 '12 at 09:27
  • @fireant You said `from the fundamental matrix, F. E transforms a point in one image to a point in the other.`. Are you suggesting that E can be used for transferring points from one view to another? I've looked everywhere including the Hartley-Zisserman book but have not found evidence to support that it can be used for transforming point coordinates between two views. Am I missing something? Could you please elaborate? – Maghoumi Mar 31 '17 at 16:13
  • @M2X I simply meant the definition of the [Essential matrix](https://en.wikipedia.org/wiki/Essential_matrix) . Sorry I'm not sure what specifically asking. – fireant Apr 09 '17 at 23:30
  • @fireant So what I was asking is this: is it possible to compute the coordinate transformation between the two images using the essential matrix? For example what coordinate in image 2 is equivalent to the point at coordinate (say) (50, 100) in image 1... I think that's what OP was asking too... – Maghoumi Apr 10 '17 at 00:46
2

Given:

Point correspondences a in image 1.

Goal:

Finding corresponding points b laying on the so called epipolar line L in image 2.

How?

    | x0 |       | x1 |
a = | y0 | , b = | y1 |
    | 1  |       | 1  |

L = F * a

    |F00 F01 F02|
F = |F10 F11 F12|
    |F20 F21 F22|

The following equation must be fulfilled to obtain b in image 2:

a' * F * b = 0.

Note: a' = transpose(a).

Färid Alijani
  • 428
  • 4
  • 16
Marina Tolkachov
  • 51
  • 3
0

For some reason I could not add a comment due to a lack of reputation. I have been studying this field for about a month now and hopefully I can answer the many questions left unanswered that have also puzzled me when I was studying the topic.

@M2X A fundamental matrix is a mapping from a point in image plane 1 to a line in image plane 2. The lines are a special type of lines called epipolar lines and are formed by the intersection of the image plane and the plane constructed from the origin of the 2 cameras and the 3D point. So it is not possible to determine a point-point mapping using the Fundamental matrix unless you have some additional information or constraints.

@Jukurrpa A homography is a point to point mapping such that parallel lines map to parallel lines. One can prove that this mapping is linear, then since linear maps a equivalent to matrices, the homography can be defined by a matrix. A set of 3D points lying on a plane projected to the image plane maps parallel lines to parallel lines, so a homography will work in your case. Methods of estimating homograph from a given set of points is outlined in the book (multiple view geometry in computer vision). Given corresponding points in both images you can find homography by using iterative approaches (Gradient Descent) or closed form solutions (Singular Value Decomposition).

ychnh
  • 157
  • 1
  • 11
  • Im confused. If the fundamental matrix IS a mapping of points in image one to image2, why can't you just take the dot product of the fundamental matrix and any point, and you will get its corresponding point in the other image? – john ktejik Nov 24 '19 at 00:18
  • Fundamental matrix a mappting from a **point** in image 1 to a **line** in image 2. In the FIGURE linked, x' maps to l'' and x'' maps to l'. [FIGURE](https://i.pinimg.com/originals/de/61/bc/de61bc8c31ee344c1fe0aaa990dd1732.jpg) – ychnh Nov 25 '19 at 01:20
  • Since x'' * l'' = 0 (this is a property of homogenous lines and points) and l'' = Fx' we see that x''Fx' = 0 for all sets of corresponding points. So if you can find a good estimation of F for your scene, given F and x' you can estimate x'' by finding a value x'' that minimizes x''Fx' to be close to 0. Typical methods of estimation involve least square regression which is equivalent to SVD (From what I have read, have not proved this myself) (Singular value decomposition) – ychnh Nov 25 '19 at 01:27
  • according to wikipedia the fundamental matrix maps points to points. "the fundamental matrix is a 3×3 matrix which relates corresponding points" – john ktejik Nov 25 '19 at 05:20
  • Yes. It does **relate** corresponding points. But it is not a 1-1 mapping. . . " fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images. In epipolar geometry, with homogeneous image coordinates, x and x′, of corresponding points in a stereo image pair, **Fx describes a line (an epipolar line) on which the corresponding point x′ on the other image must lie.** " (Wikipedia) – ychnh Nov 25 '19 at 05:24
  • ok thanks. This explains a lot on why i can't get my project working. I've just been applying the fundamental matrix to points and expecting the resulting points, and trying to triangulate them... yeah... back to square zero for me – john ktejik Nov 25 '19 at 05:28
  • You might be interested in the homography matrix. It gives you a point-point mapping. However, there are some restrictions in a homography in that it can only describe image to image mappings that keep parallel lines parallel from image 1 to image 2. From my understanding, recent 3D reconstruction methods 'OrbSlam' employ both methods 'Homography matrix and Fundamental matrix'. **Another** point that might be helpful is to normalize your data. **Finally** if you want to triangulate the points in 3D space, you must recover motion, rotation/trnslation from the H/F matrix, which is bit hard – ychnh Nov 25 '19 at 05:33
  • but is outlined in the references of the paper (OrbSlam) https://arxiv.org/abs/1502.00956 – ychnh Nov 25 '19 at 05:36
  • 'you must recover motion, rotation/trnslation from the H/F matrix'. Yeah thats what I need to do. I thought the F matrix *was* the rotation/translation matrices – john ktejik Nov 25 '19 at 05:57
  • The paper seems cool, but unfortunately it says it needs the K (camera calibration) matrices, which I do not have. – john ktejik Nov 25 '19 at 06:01