Picking Ray is inaccurate

Question

I'm trying to implement a picking ray via instructions from this website.

Right now I basically only want to be able to click on the ground to order my little figure to walk towards this point. Since my ground plane is flat , non-rotated and non-translated I'd have to find the x and z coordinate of my picking ray when y hits 0.

So far so good, this is what I've come up with:

//some constants
float HEIGHT = 768.f;
float LENGTH = 1024.f;
float fovy = 45.f;
float nearClip = 0.1f;

//mouse position on screen
float x = MouseX;
float y = HEIGHT - MouseY;

//GetView() returns the viewing direction, not the lookAt point.
glm::vec3 view = cam->GetView();
glm::normalize(view);

glm::vec3 h = glm::cross(view, glm::vec3(0,1,0) ); //cameraUp
glm::normalize(h);

glm::vec3 v = glm::cross(h, view);
glm::normalize(v);

// convert fovy to radians 
float rad = fovy * 3.14 / 180.f; 
float vLength = tan(rad/2) * nearClip; //nearClippingPlaneDistance
float hLength = vLength * (LENGTH/HEIGHT);

v *= vLength;
h *= hLength;

// translate mouse coordinates so that the origin lies in the center
// of the view port
x -= LENGTH / 2.f;
y -= HEIGHT / 2.f;

// scale mouse coordinates so that half the view port width and height
// becomes 1
x /= (LENGTH/2.f);
y /= (HEIGHT/2.f);

glm::vec3 cameraPos = cam->GetPosition();

// linear combination to compute intersection of picking ray with
// view port plane
glm::vec3 pos = cameraPos + (view*nearClip) + (h*x) + (v*y);

// compute direction of picking ray by subtracting intersection point
// with camera position
glm::vec3 dir = pos - cameraPos;

//Get intersection between ray and the ground plane
pos -= (dir * (pos.y/dir.y));

At this point I'd expect pos to be the point where my picking ray hits my ground plane. When I try it, however, I get something like this: (The mouse cursor wasn't recorded) It's hard to see since the ground has no texture, but the camera is tilted, like in most RTS games. My pitiful attempt to model a remotely human looking being in Blender marks the point where the intersection happened according to my calculation.

So it seems that the transformation between view and dir somewhere messed up and my ray ended up pointing in the wrong direction. The gap between the calculated position and the actual position increases the farther I mouse my move away from the center of the screen.

I've found out that:

• HEIGHT and LENGTH aren't acurate. Since Windows cuts away a few pixels for borders it'd be more accurate to use 1006,728 as window resolution. I guess that could make for small discrepancies.
• If I increase fovy from 45 to about 78 I get a fairly accurate ray. So maybe there's something wrong with what I use as fovy. I'm explicitely calling glm::perspective(45.f, 1.38f, 0.1f, 500.f) (fovy, aspect ratio, fNear, fFar respectively).

So here's where I am lost. What do I have to do in order to get an accurate ray?

PS: I know that there are functions and libraries that have this implemented, but I try to stay away from these things for learning purposes.

Answer 1

Here's working code that does cursor to 3D conversion using depth buffer info:

  glGetIntegerv(GL_VIEWPORT, @fViewport);
  glGetDoublev(GL_PROJECTION_MATRIX, @fProjection);
  glGetDoublev(GL_MODELVIEW_MATRIX, @fModelview);

  //fViewport already contains viewport offsets
  PosX := X;
  PosY := ScreenY - Y; //In OpenGL Y axis is inverted and starts from bottom

  glReadPixels(PosX, PosY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, @vz);

  gluUnProject(PosX, PosY, vz, fModelview, fProjection, fViewport, @wx, @wy, @wz);
  XYZ.X := wx;
  XYZ.Y := wy;
  XYZ.Z := wz;

If you do test only ray/plane intersection this is the second part without DepthBuffer:

  gluUnProject(PosX, PosY, 0, fModelview, fProjection, fViewport, @x1, @y1, @z1); //Near
  gluUnProject(PosX, PosY, 1, fModelview, fProjection, fViewport, @x2, @y2, @z2); //Far

  //No intersection
  Result := False;
  XYZ.X := 0;
  XYZ.Y := 0;
  XYZ.Z := aZ;

  if z2 < z1 then
    SwapFloat(z1, z2);

  if (z1 <> z2) and InRange(aZ, z1, z2) then
  begin
    D := 1 - (aZ - z1) / (z2 - z1);
    XYZ.X := Lerp(x1, x2, D);
    XYZ.Y := Lerp(y1, y2, D);
    Result := True;
  end;

I find it rather different from what you are doing, but maybe that will make more sense.