Using warpPerspective() on a sequence of points given by HoughCircles(), OpenCV

Question

I'm trying to detect the positions of billiards balls on a table from an image taken at a perspective angle. I'm using the getPerspectiveTransform() method to find the transformation matrix and I want to apply that to only the circles I detect using HoughCircles. I'm trying to go from a rather large trapezoidal shape to a smaller rectangular shape. I don't want to do the transformation on the image first and then find the HoughCircles because the image gets too warped for houghcircles to provide useful results.

Here's my code:

        CvMat mmat = cvCreateMat(3,3,CV_32FC1);
        double srcX1 = 462;
        double srcX2 = 978;
        double srcX3 = 1440;
        double srcX4 = 0;
        double srcY = 241;
        double srcHeight = 772;

        double dstX = 56.8;
        double dstY = 33.5;
        double dstWidth = 262.4;
        double dstHeight = 447.3;

        CvSeq seq = cvHoughCircles(newGray, circles, CV_HOUGH_GRADIENT, 2.1d, (double)newGray.height()/40, 85d, 65d, 5, 50);

        JavaCV.getPerspectiveTransform(new double[]{srcX1, srcY, srcX2,srcY, srcX3, srcHeight, srcX4, srcHeight}, 
                  new double[]{dstX, dstY, dstWidth, dstY, dstWidth, dstHeight, dstX, dstHeight}, mmat);
        cvWarpPerspective(seq, seq, mmat);


        for(int j=0; j<seq.total(); j++){
            CvPoint3D32f point = new CvPoint3D32f(cvGetSeqElem(seq, j));

            float xyr[] = {point.x(),point.y(),point.z()};
            CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));

            int radius = Math.round(xyr[2]);
            cvCircle(gray, center, 3, CvScalar.GREEN, -1, 8, 0);
            cvCircle(gray, center, radius, CvScalar.BLUE, 3, 8, 0);
        }

The problem is I get this error on the warpPerspective() method:

error: (-215) seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size in function cv::Mat cv::cvarrToMat(const CvArr*, bool, bool, int) 

Also I guess it's worth mentioning that I'm using JavaCV, in case the method calls look a bit different than what you're used to. Thanks for any help.

Answer 1

Answer:

the problem with what you want to do (besides the obvious, opencv wont let you) is that the radius cant really be warped correctly. AFAIK the xy coordinates are pretty easy to calculate x'=((m00x+m01y+m02)/(m20x+m21y+m22)) y'=((m10x+m11y+m12)/(m20x+m21y_m22)) when m is the transformation matrix. the radius you can hack by transforming all the points of the original circle and then find the max distance between x'y' and those points (atleast if the radius in the warped image is expected to cover all those points) btw, mIJx = m(i,j)*x (just to clarify)

End Answer.

---

Everything i write is according to the c++ version, i've never used JavaCV but from what i could see its just a wrapper that calls the native c++ lib.

CvSeq is a sequance data structure that behaves like a linked list. the assert your application crushes at is

CV_Assert(seq->total > 0 && CV_ELEM_SIZE(seq->flags) == seq->elem_size);

which means that either your seq instance is empty (total is the number of elements in the sequence) or somehow the inner seq flags are corrupted.

I'd recommend that you'd check the total member of your CvSeq, and the cvHoughCircles call. all of this occurs before the actual implementation of cvWarpPerspective (its the first line in the implementation, that only converts your CvSeq to cv::Mat).. so its not the warping but what you're doing before that. anyway, to understand whats wrong with cvHoughCircles we'll need more info about the creation of newGray and circles.

here is an example i've found on the javaCV page (Link)

IplImage gray = cvCreateImage( cvSize( img.width, img.height ), IPL_DEPTH_8U, 1 );
cvCvtColor( img, gray, CV_RGB2GRAY );
// smooth it, otherwise a lot of false circles may be detected
cvSmooth(gray,gray,CV_GAUSSIAN,9,9,2,2);
CvMemStorage circles = CvMemStorage.create();
CvSeq seq = cvHoughCircles(gray, circles.getPointer(), CV_HOUGH_GRADIENT,
                                                2, img.height/4, 100, 100, 0, 0);
for(int i=0; i<seq.total; i++){
        float xyr[] = cvGetSeqElem(seq,i).getFloatArray(0, 3);
        CvPoint center = new CvPoint(Math.round(xyr[0]), Math.round(xyr[1]));

        int radius = Math.round(xyr[2]);
        cvCircle(img, center.byValue(), 3, CvScalar.GREEN, -1, 8, 0);
        cvCircle(img, center.byValue(), radius, CvScalar.BLUE, 3, 8, 0);
}

from what i've seen in the implementation of cvHoughCircles, the answer is saved in the circles buff and at the end they create from it the CvSeq to return, so if you've allocated the circles buff wrong, it wont work.

EDIT:

as you can see, the CvSeq instance in case of the return from cvHoughCircles is a list of point-values, that is probably why the assertion failed. you cannot convert this CvSeq into a cv::Mat.. because its just not a cv::Mat. to get only the circles returned from cvHoughCircles in an cv::Mat instance, you'll need to create a new cv::Mat instance and than draw onto it all the circles in the CvSeq - as seen in the provided example above. than the warping will work (you'll have a cv::Mat instance, and that is what the function expect - a cv::Mat as the only element in the CvSeq)

END EDIT

here is the c++ reference for CvSeq and if you want to fiddle with the source code than

cvarrToMat is in matrix.cpp

CV_ELEM_SIZE is in types_c.h

cvWarpPerspective is in imgwarp.cpp

cvHoughCircles is in hough.cpp

I hope that will help.

BTW, your next error will probably be: cv::warpPerspective in the C++ OpencCv asserts that dst.data != src.data

thus

cvWarpPerspective(seq, seq, mmat);

wont work cause your source mat and destination mat referencing the same data. Not all the functions in OpenCV (and image processing in general) work in-situ (because there is no in-situ algorithm or because its slower then the other version eg. transpose of an n*n mat will work in-situ, but n*m where n!=m will be harder to do in-situ and might be slower) you cant assume the using the src matrix as the dst will work.