Detecting edges of a card with rounded corners

Question

Hi currently i am working on an OCR reading app where i have successfully able to capture the card image by using AVFoundation framework.

For next step, i need to find out edges of the card , so that i can crop the card image from main captured image & later i can sent it to OCR engine for processing.

The main problem is now to find the edges of the card & i am using below code(taken from another open source project) which uses OpenCV for this purpose.It is working fine if the card is pure rectangular Card or Paper. But when i use a card with rounded corner (e.g Driving License), it is failed to detect . Also i dont have much expertise in OpenCV , Can any one help me in solving this issue?

- (void)detectEdges
{
    cv::Mat original = [MAOpenCV cvMatFromUIImage:_adjustedImage];
    CGSize targetSize = _sourceImageView.contentSize;
    cv::resize(original, original, cvSize(targetSize.width, targetSize.height));

    cv::vector<cv::vector<cv::Point>>squares;
    cv::vector<cv::Point> largest_square;

    find_squares(original, squares);
    find_largest_square(squares, largest_square);

    if (largest_square.size() == 4)
    {

        // Manually sorting points, needs major improvement. Sorry.

        NSMutableArray *points = [NSMutableArray array];
        NSMutableDictionary *sortedPoints = [NSMutableDictionary dictionary];

        for (int i = 0; i < 4; i++)
        {
            NSDictionary *dict = [NSDictionary dictionaryWithObjectsAndKeys:[NSValue valueWithCGPoint:CGPointMake(largest_square[i].x, largest_square[i].y)], @"point" , [NSNumber numberWithInt:(largest_square[i].x + largest_square[i].y)], @"value", nil];
            [points addObject:dict];
        }

        int min = [[points valueForKeyPath:@"@min.value"] intValue];
        int max = [[points valueForKeyPath:@"@max.value"] intValue];

        int minIndex;
        int maxIndex;

        int missingIndexOne;
        int missingIndexTwo;

        for (int i = 0; i < 4; i++)
        {
            NSDictionary *dict = [points objectAtIndex:i];

            if ([[dict objectForKey:@"value"] intValue] == min)
            {
                [sortedPoints setObject:[dict objectForKey:@"point"] forKey:@"0"];
                minIndex = i;
                continue;
            }

            if ([[dict objectForKey:@"value"] intValue] == max)
            {
                [sortedPoints setObject:[dict objectForKey:@"point"] forKey:@"2"];
                maxIndex = i;
                continue;
            }

            NSLog(@"MSSSING %i", i);

            missingIndexOne = i;
        }

        for (int i = 0; i < 4; i++)
        {
            if (missingIndexOne != i && minIndex != i && maxIndex != i)
            {
                missingIndexTwo = i;
            }
        }


        if (largest_square[missingIndexOne].x < largest_square[missingIndexTwo].x)
        {
            //2nd Point Found
            [sortedPoints setObject:[[points objectAtIndex:missingIndexOne] objectForKey:@"point"] forKey:@"3"];
            [sortedPoints setObject:[[points objectAtIndex:missingIndexTwo] objectForKey:@"point"] forKey:@"1"];
        }
        else
        {
            //4rd Point Found
            [sortedPoints setObject:[[points objectAtIndex:missingIndexOne] objectForKey:@"point"] forKey:@"1"];
            [sortedPoints setObject:[[points objectAtIndex:missingIndexTwo] objectForKey:@"point"] forKey:@"3"];
        }


        [_adjustRect topLeftCornerToCGPoint:[(NSValue *)[sortedPoints objectForKey:@"0"] CGPointValue]];
        [_adjustRect topRightCornerToCGPoint:[(NSValue *)[sortedPoints objectForKey:@"1"] CGPointValue]];
        [_adjustRect bottomRightCornerToCGPoint:[(NSValue *)[sortedPoints objectForKey:@"2"] CGPointValue]];
        [_adjustRect bottomLeftCornerToCGPoint:[(NSValue *)[sortedPoints objectForKey:@"3"] CGPointValue]];
    }

    original.release();


}

Answer 1

This naive implementation is based on some of the techniques demonstrated in squares.cpp, available in the OpenCV sample directory. The following posts also discuss similar applications:

• OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection
• Square detection doesn't find squares
• Find corner of papers

@John, the code below has been tested with the sample image you provided and another one I created:

The processing pipeline starts with findSquares(), a simplification of the same function implemented by OpenCV's squares.cpp demo. This function converts the input image to grayscale and applies a blur to improve the detection of the edges (Canny):

The edge detection is good, but a morphological operation (dilation) is needed to join nearby lines:

After that we try to find the contours (edges) and assemble squares out of them. If we tried to draw all the detected squares on the input images, this would be the result:

It looks good, but it's not exactly what we are looking for since there are too many detected squares. However, the largest square is actually the card, so from here on it's pretty simple and we just figure out which of the squares is the largest. That's exactly what findLargestSquare() does.

Once we know the largest square, we simply paint red dots at the corners of the square for debugging purposes:

As you can see, the detection is not perfect but it seems good enough for most uses. This is not a robust solution and I only wanted to share one approach to solve the problem. I'm sure that there are other ways to deal with this that might be more interesting to you. Good luck!

#include <iostream>
#include <cmath>
#include <vector>

#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/imgproc/imgproc_c.h>

/* angle: finds a cosine of angle between vectors, from pt0->pt1 and from pt0->pt2
 */
double angle(cv::Point pt1, cv::Point pt2, cv::Point pt0)
{
    double dx1 = pt1.x - pt0.x;
    double dy1 = pt1.y - pt0.y;
    double dx2 = pt2.x - pt0.x;
    double dy2 = pt2.y - pt0.y;
    return (dx1*dx2 + dy1*dy2)/sqrt((dx1*dx1 + dy1*dy1)*(dx2*dx2 + dy2*dy2) + 1e-10);
}

/* findSquares: returns sequence of squares detected on the image
 */
void findSquares(const cv::Mat& src, std::vector<std::vector<cv::Point> >& squares)
{
    cv::Mat src_gray;
    cv::cvtColor(src, src_gray, cv::COLOR_BGR2GRAY);

    // Blur helps to decrease the amount of detected edges
    cv::Mat filtered;
    cv::blur(src_gray, filtered, cv::Size(3, 3));
    cv::imwrite("out_blur.jpg", filtered);

    // Detect edges
    cv::Mat edges;
    int thresh = 128;
    cv::Canny(filtered, edges, thresh, thresh*2, 3);
    cv::imwrite("out_edges.jpg", edges);

    // Dilate helps to connect nearby line segments
    cv::Mat dilated_edges;
    cv::dilate(edges, dilated_edges, cv::Mat(), cv::Point(-1, -1), 2, 1, 1); // default 3x3 kernel
    cv::imwrite("out_dilated.jpg", dilated_edges);

    // Find contours and store them in a list
    std::vector<std::vector<cv::Point> > contours;
    cv::findContours(dilated_edges, contours, cv::RETR_LIST, cv::CHAIN_APPROX_SIMPLE);

    // Test contours and assemble squares out of them
    std::vector<cv::Point> approx;
    for (size_t i = 0; i < contours.size(); i++)
    {
        // approximate contour with accuracy proportional to the contour perimeter
        cv::approxPolyDP(cv::Mat(contours[i]), approx, cv::arcLength(cv::Mat(contours[i]), true)*0.02, true);

        // Note: absolute value of an area is used because
        // area may be positive or negative - in accordance with the
        // contour orientation
        if (approx.size() == 4 && std::fabs(contourArea(cv::Mat(approx))) > 1000 &&
            cv::isContourConvex(cv::Mat(approx)))
        {
            double maxCosine = 0;
            for (int j = 2; j < 5; j++)
            {
                double cosine = std::fabs(angle(approx[j%4], approx[j-2], approx[j-1]));
                maxCosine = MAX(maxCosine, cosine);
            }

            if (maxCosine < 0.3)
                squares.push_back(approx);
        }
    }
}

/* findLargestSquare: find the largest square within a set of squares
 */
void findLargestSquare(const std::vector<std::vector<cv::Point> >& squares,
                       std::vector<cv::Point>& biggest_square)
{
    if (!squares.size())
    {
        std::cout << "findLargestSquare !!! No squares detect, nothing to do." << std::endl;
        return;
    }

    int max_width = 0;
    int max_height = 0;
    int max_square_idx = 0;
    for (size_t i = 0; i < squares.size(); i++)
    {
        // Convert a set of 4 unordered Points into a meaningful cv::Rect structure.
        cv::Rect rectangle = cv::boundingRect(cv::Mat(squares[i]));

        //std::cout << "find_largest_square: #" << i << " rectangle x:" << rectangle.x << " y:" << rectangle.y << " " << rectangle.width << "x" << rectangle.height << endl;

        // Store the index position of the biggest square found
        if ((rectangle.width >= max_width) && (rectangle.height >= max_height))
        {
            max_width = rectangle.width;
            max_height = rectangle.height;
            max_square_idx = i;
        }
    }

    biggest_square = squares[max_square_idx];
}

int main()
{
    cv::Mat src = cv::imread("cc.png");
    if (src.empty())
    {
        std::cout << "!!! Failed to open image" << std::endl;
        return -1;
    }

    std::vector<std::vector<cv::Point> > squares;
    findSquares(src, squares);

    // Draw all detected squares
    cv::Mat src_squares = src.clone();
    for (size_t i = 0; i < squares.size(); i++)
    {
        const cv::Point* p = &squares[i][0];
        int n = (int)squares[i].size();
        cv::polylines(src_squares, &p, &n, 1, true, cv::Scalar(0, 255, 0), 2, CV_AA);
    }
    cv::imwrite("out_squares.jpg", src_squares);
    cv::imshow("Squares", src_squares);

    std::vector<cv::Point> largest_square;
    findLargestSquare(squares, largest_square);

    // Draw circles at the corners
    for (size_t i = 0; i < largest_square.size(); i++ )
        cv::circle(src, largest_square[i], 4, cv::Scalar(0, 0, 255), cv::FILLED);
    cv::imwrite("out_corners.jpg", src);

    cv::imshow("Corners", src);
    cv::waitKey(0);

    return 0;
}

Answer 2

instead of "pure" rectangular blobs, try to go for nearly rectangular ones.

1- gaussian blur

2- grayscale and canny edge detection

3- extract all blobs (contours) in your image and filter out small ones. you will use findcontours and contourarea functions for that purpose.

4- using moments, filter out non-rectangular ones. First you need to check out moments of rectangle-like objects. You can do it by yourself or google it. Then list those moments and find similarity between objects, create your filter as such.

Ex: After test, say you found out central moment m30's are similar for rectangle-like objects -> filter out objects having inaccurate m30.

Answer 3

I know maybe it's too late for this post, but I am posting this in case it might help someone else.

The iOS Core Image framework already has a good tool to detect features such as rectangles (since iOS 5), faces, QR codes and even regions containing text in a still image. If you check out the CIDetector class you'll find what you need. I am using it for an OCR app too, it's super easy and very reliable compared to what you can do with OpenCV (I am not good with OpenCV, but the CIDetector gives much better results with 3-5 lines of code).

Answer 4

I don't know if it is an option, but you could have the user define the edges of it rather than trying to do it programatically.