OpenGL ES 2.0 Object Picking on iOS

Question

What is the best method to select objects that have been drawn in OpenGL ES 2.0 (iOS)?

I am drawing points.

Answer 1

Here is working prototype of color picking, tested on most old ipads and working well. This is actually some part of project called InCube Chess that one may find in app store. The main code you will see is located in a class derived from GLKViewController like this:

@interface IncubeViewController : GLKViewController

This means you have glkview in it: ((GLKView *)self.view).

Here are also some properties:

@property (strong, nonatomic) EAGLContext *context;
@property (strong, nonatomic) GLKBaseEffect *effect;

Don't forget to synthesize them in your *.m file.

@synthesize context = _context;
@synthesize effect = _effect;

The idea is that you have chess pieces on your table (or some objects in your 3d scene) and you need to find a piece in your list of pieces by tapping on a screen. That is, you need to convert your 2d screen tap coords (@point in this case) to chess piece instance.

Each piece has its unique id that I call a "seal". You can allocate the seals from from 1 up to something. Selection function returns piece seal found by tap coords. Then having the seal you can easily find your piece in pieces hash table or array in a way like this:

-(Piece *)findPieceBySeal:(GLuint)seal
{
        /* !!! Black background in off screen buffer produces 0 seals. This allows
           to quickly filter out taps that did not select anything (will be
           mentioned below) !!! */
        if (seal == 0)
                return nil;
        PieceSeal *sealKey = [[PieceSeal alloc] init:s];
        Piece *p = [sealhash objectForKey:sealKey];
        [sealKey release];
        return p;
}

"sealhash" is a NSMutableDictionary.

Now this is the main selection function. Note, that my glkview is antialised and you can't use its buffers for color picking. This mean you need to create your own off screen buffer with antialiasing disabled for picking purposes only.

- (NSUInteger)findSealByPoint:(CGPoint)point
{
        NSInteger height = ((GLKView *)self.view).drawableHeight;
        NSInteger width = ((GLKView *)self.view).drawableWidth;
        Byte pixelColor[4] = {0,};
        GLuint colorRenderbuffer;
        GLuint framebuffer;

        glGenFramebuffers(1, &framebuffer);
        glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
        glGenRenderbuffers(1, &colorRenderbuffer);
        glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

        glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, width, height);
        glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER, colorRenderbuffer);

        GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
        if (status != GL_FRAMEBUFFER_COMPLETE) {
                NSLog(@"Framebuffer status: %x", (int)status);
                return 0;
        }

        [self render:DM_SELECT];

        CGFloat scale = UIScreen.mainScreen.scale;
        glReadPixels(point.x * scale, (height - (point.y * scale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);

        glDeleteRenderbuffers(1, &colorRenderbuffer);
        glDeleteFramebuffers(1, &framebuffer);

        return pixelColor[0];
}

Note that function takes into account display scale (retina or new iPads).

Here is render() function used in function above. Note, that for rendering purposes it clear s the buffer with some background color and for selecting case it makes it black so that you can easily check if you tapped on any piece at all or not.

- (void) render:(DrawMode)mode
{
        if (mode == DM_RENDER)
                glClearColor(backgroundColor.r, backgroundColor.g,
                             backgroundColor.b, 1.0f);
        else
                glClearColor(0.0f, 0.0f, 0.0f, 1.0f);

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        /* Draw all pieces. */
        for (int i = 0; i < [model->pieces count]; i++) {
                Piece *p = [model->pieces objectAtIndex:i];
                [self drawPiece:p mode:mode];
        }
}

Next is how we draw the piece.

- (void) drawPiece:(Piece *)p mode:(DrawMode)mode
{
        PieceType type;

        [self pushMatrix];

        GLKMatrix4 modelViewMatrix = self.effect.transform.modelviewMatrix;

        GLKMatrix4 translateMatrix = GLKMatrix4MakeTranslation(p->drawPos.X,
                                                               p->drawPos.Y,
                                                               p->drawPos.Z);
        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, translateMatrix);

        GLKMatrix4 rotateMatrix;
        GLKMatrix4 scaleMatrix;

        if (mode == DM_RENDER) {
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X,
                                                  p->scale.Y, p->scale.Z);
        } else {
                /* !!! Make the piece a bit bigger in off screen buffer for selection
                   purposes so that we always sure that we tapped it correctly by
                   finger.*/
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X + 0.2,
                                                  p->scale.Y + 0.2, p->scale.Z + 0.2);
        }

        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, scaleMatrix);

        self.effect.transform.modelviewMatrix = modelViewMatrix;

        type = p->type;

        if (mode == DM_RENDER) {
                /* !!! Use real pieces color and light on for normal drawing !!! */
                GLKVector4 color[pcLast] = {
                        [pcWhite] = whitesColor,
                        [pcBlack] = blacksColor
                };
                self.effect.constantColor = color[p->color];
                self.effect.light0.enabled = GL_TRUE;
        } else {
                /* !!! Use piece seal for color. Important to turn light off !!! */
                self.effect.light0.enabled = GL_FALSE;
                self.effect.constantColor = GLKVector4Make(p->seal / 255.0f,
                                                           0.0f, 0.0f, 0.0f);
        }

        /* Actually normal render the piece using it geometry buffers. */
        [self renderPiece:type];

        [self popMatrix];
}

This is how to use the functions shown above.

- (IBAction) tapGesture:(id)sender
{
        if ([(UITapGestureRecognizer *)sender state] == UIGestureRecognizerStateEnded) {
                CGPoint tap = [(UITapGestureRecognizer *)sender locationInView:self.view];
                Piece *p = [self findPieceBySeal:[self findSealByPoint:tap]];

                /* !!! Do something with your selected object !!! */
        }
}

This is basically it. You will have very precise picking algorithm that is much better than ray tracing or others.

Here helpers for push/pop matrix things.

- (void)pushMatrix
{
        assert(matrixSP < sizeof(matrixStack) / sizeof(GLKMatrix4));
        matrixStack[matrixSP++] = self.effect.transform.modelviewMatrix;
}

- (void)popMatrix
{
        assert(matrixSP > 0);
        self.effect.transform.modelviewMatrix = matrixStack[--matrixSP];
}

Here also glkview setup/cleanup functions that I used.

- (void)viewDidLoad
{
        [super viewDidLoad];
        self.context = [[[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2] autorelease];
        if (!self.context)
                NSLog(@"Failed to create ES context");

        GLKView *view = (GLKView *)self.view;
        view.context = self.context;
        view.drawableDepthFormat = GLKViewDrawableDepthFormat24;

        [self setupGL];
}

- (void)viewDidUnload
{    
        [super viewDidUnload];

        [self tearDownGL];

        if ([EAGLContext currentContext] == self.context)
                [EAGLContext setCurrentContext:nil];
        self.context = nil;
}

- (void)setupGL
{
        [EAGLContext setCurrentContext:self.context];

        self.effect = [[[GLKBaseEffect alloc] init] autorelease];
        if (self.effect) {
                self.effect.useConstantColor = GL_TRUE;
                self.effect.colorMaterialEnabled = GL_TRUE;
                self.effect.light0.enabled = GL_TRUE;
                self.effect.light0.diffuseColor = GLKVector4Make(1.0f, 1.0f, 1.0f, 1.0f);
        }

        /* !!! Draw antialiased geometry !!! */
        ((GLKView *)self.view).drawableMultisample = GLKViewDrawableMultisample4X;
        self.pauseOnWillResignActive = YES;
        self.resumeOnDidBecomeActive = YES;
        self.preferredFramesPerSecond = 30;

        glDisable(GL_DITHER);
        glEnable(GL_CULL_FACE);
        glEnable(GL_DEPTH_TEST);
        glLineWidth(2.0f);

        /* Load pieces geometry */
        [self loadGeometry];
}

- (void)tearDownGL
{
        drawReady = NO;
        [EAGLContext setCurrentContext:self.context];
        [self unloadGeometry];
}

Hope this helps and may be closes "the picking question" forever :)

Answer 2

Here is how i do, based on the above solution, with depth buffer for 3D picking and GLKVector3 signature to retrieve the seal :

• all my objects got a seal
• i use a GLKVector3 named color to retrieved the seal from the read pixel. With this solution you can store 255 *255 * 255 = 16,581,375 objects.

In the Shader (vertex or fragment as you want)

add a picking boolean which tell you if you do the pick pass or not

uniform bool picking;

and the GLKVector3

uniform vec3 color;

if the boolean picking is activated so the color will be the GLKVector3

if(picking)
{
    colorVarying = vec4(color, 1.0);
}

In the ViewController

the method which create the GLKVector3 from a seal (when i create a new 3D object) :

-(GLKVector3)pack:(uint)seal
{
    GLKVector3 hash;

    float r = seal % 255;
    float g = (seal / 255) % 255;
    float b = (seal / (255 * 255)) % 255;
    hash = GLKVector3Make(r/255, g/255, b/255);

    return hash;
}

And the view controller code where you get the pixel from the touch position and get the selected seal from the selected color :

-(uint)getSealByColor:(GLKVector3)color
{
    color = GLKVector3DivideScalar(color, 255);
    for (MyObject *o in _objects) {
        if(GLKVector3AllEqualToVector3(o.color, color))
        {
            return o.seal;
        }
    }
    return 0;
}

-(void)tap:(UITapGestureRecognizer*)recognizer
{
    CGPoint p = [recognizer locationInView:self.view];
    GLKVector3 i = [self pickingAt:p];
    _sealSelected = [self getSealByColor:i];
}

-(GLKVector3)pickingAt:(CGPoint)position
{
    CGFloat scale = [UIScreen mainScreen].scale;

    GLsizei w = self.view.bounds.size.width * scale;
    GLsizei h = self.view.bounds.size.height * scale;

    GLuint fb;
    GLuint rb;
    GLuint db;

    Byte pixelColor[4] = {0,};

    glGenFramebuffers(1, &fb);
    glBindFramebuffer(GL_FRAMEBUFFER, fb);
    glGenRenderbuffers(1, &rb);
    glBindRenderbuffer(GL_RENDERBUFFER, rb);

    glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, w, h);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, rb);

    //here we also create a depth buffer for 3D objects picking
    glGenRenderbuffers(1, &db);
    glBindRenderbuffer(GL_RENDERBUFFER, db);

    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, w, h);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, db);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    if (status != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"Framebuffer status: %x", (int)status);
        return GLKVector3Make(0.0, 0.0, 0.0);
    }

    //we render the scene with our picking boolean activated
    [self render:YES];

    glReadPixels(position.x * scale, (h - (position.y * scale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);

    glDeleteRenderbuffers(1, &db);
    glDeleteRenderbuffers(1, &rb);
    glDeleteFramebuffers(1, &fb);

    return GLKVector3Make(pixelColor[0], pixelColor[1], pixelColor[2]);
}

Hope this help someone, this is how i do to allow color picking for more than 255 objects.