Square detection with aberrations in edges and 1 missing corner

Question

This is a follow up to couple of similar questions regarding square detection in which karlphillip, mevatron, and abid-rahman-k came up with some cool approaches.

I am trying to design a robust square detection algorithm to help isolate a picture of a receipt from the rest of the image. My code is built off the convex hull approach from the previous questions but it's choking on an image where one of the points isn't in the image and the edges of the receipt have aberrations due to a pen holder on the left side.

How can I detect the corners on this receipt?

Here is the image:

image of receipt

Here is my code:

import cv2
import numpy as np

img = cv2.imread('taco.jpg')
img = cv2.resize(img,(1944,2592))
img = cv2.medianBlur(img,31)
img = cv2.GaussianBlur(img,(0,0),3)

grayscale = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
thresh = cv2.Canny(grayscale, 10, 20)
thresh = cv2.dilate(thresh,None)

contours,hier = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)

for cnt in contours:
    if cv2.contourArea(cnt)>250:  # remove small areas like noise etc
        hull = cv2.convexHull(cnt)    # find the convex hull of contour
        hull = cv2.approxPolyDP(hull,0.1*cv2.arcLength(hull,True),True)
        if len(hull)==4:
            cv2.drawContours(img,[hull],0,(0,255,0),2)

cv2.namedWindow('output',cv2.cv.CV_WINDOW_NORMAL)
cv2.imshow('output',img)
cv2.cv.ResizeWindow('output',960,640)
cv2.waitKey()
cv2.destroyAllWindows()

Any ideas?

score 3 · Accepted Answer · answered Sep 04 '12 at 17:12

3

A solution in Mathematica:

Import your image:

i = Import@"http://i.imgur.com/RrYKJ.jpg";

Detect edges in a scale grater than the letters in the receipt (a parameter)

i1 = EdgeDetect[i, 10]

Mathematica graphics

Delete lines smaller than the scale of the perimeter of the receipt (a parameter)

i2 = DeleteSmallComponents[i1, 1000]

Mathematica graphics

Find the morphological components

(mc = MorphologicalComponents[Erosion[ColorNegate@i2, 1]]) // Colorize

Mathematica graphics

Find the morph component with more border adjacency (to delete it from the mask)

com = Commonest[Join[mc[[1]], mc[[-1]], Transpose[mc][[1]], Transpose[mc][[-1]]]]

Form the mask

mc1 = Unitize[mc /. com[[1]] -> 0];

Mathematica graphics

Multiply the mask by your original image

ImageMultiply[Image@mc1, i]

Mathematica graphics

answered Sep 04 '12 at 17:12

Dr. belisarius

59,172
13
109
187

This looks great although I can't afford a mathematica license. However, the only step I don't know how to mimic in python & opencv is the first one. Do you know the algorithm behind: i1 = EdgeDetect[i, 10] Specifically I'm wondering how the algorithm edge detects at a scale of 10. Maybe I can downscale my image or blur it at a level of 10 to get similar edge detection? – slykat Sep 04 '12 at 17:38
Canny should work nicely here. One step that's missing in Verde's recipe is the modeling of the object's boundaries - you will likely want to robustly fit lines to them, so to estimate the four corners and remove the perspective distortion. – Francesco Callari Sep 05 '12 at 02:30
@Franco Past questions from the OP pointed in that direction. The OP needed to ask this very question, because past answers were based in a four-sided polygon. I tried to remain as agnostic as i could be wrt the geometric figure. You really don't know the number of corners (5 in my image). The perspective distortion is trivial to remove once you've mapped the ticket boundaries. – Dr. belisarius Sep 05 '12 at 02:41
@Verde, apologies if my comment sounded out as criticism of yours. It was not intended - indeed I applaud your taking the time to find and demonstrate an end-to-end answer to the OP. But, do note that we do know the expected number of corners - 4 is a prior here ;-) (OP did say he's looking for the receipt, and in my neck of the woods they are still rectangular - something to do with printers on paper rolls). 'Course paper bends, lenses have distortion etc. – Francesco Callari Sep 05 '12 at 12:58
@FrancoCallari It's OK, but note that if the photo is carelessly taken and some border of the receipt is left out (as it partially happened here), a rectangle matching algo will fail miserably. – Dr. belisarius Sep 05 '12 at 13:09
Yea, Verde is correct - there are a lot of issues that can break approaches like rectangle matching. Specifically, one of the corners could be missing, the receipt can be curled, shadows, etc. Thus, i've been struggling to find a robust algorithm. Trying hough transform & convex hull approaches but problems with both. @Verde - The main thing I'm struggling to do is implement i1 = EdgeDetect[i, 10]. It seems to have beautifully removed almost all the edges from noise and text - which I can't seem to do in opencv & python. Is this simply a canny edge detector with large aperture? Any ideas? – slykat Sep 06 '12 at 06:05
@slykat I can get almost the same result convolving first with a Gaussian filter of size 10 and then applying a gradient filter (you need to tune the cutoff). Perhaps you'll need to binarize after that. Basically `EdgeDetect` is a tuned Cany, but I don't know the internal parms – Dr. belisarius Sep 06 '12 at 06:24

Square detection with aberrations in edges and 1 missing corner

1 Answers1