How to detect robot direction from Image?

Question

I've developing the robot which can run in the corn plant and guided by compass sensors but I want to apply the camera as the eye of robot and use the image processing to detect error angle of movement.

This is the image examples.

processed image raw image segmented image

I use this following step

1. Step 1 : The current technique that I use is convert color value to HSV modified from this code

2. Step 2 : So It will detect selected color which is the brown or dirt color then I collect the most left and right of brown or selected color of each image row in two array(a red point).

3. Step 3 : I plot a 2 linear regression line as a blue point and calculate intersection point as a pink point
4. Step 4 : Draw the green line to compare the pink point with another images. I'm not sure what to do with this green line yet
5. The problem is the dirt or brown color is exist between the corn leaf too then I make my code to miss a calculation

The question is how to filter out the brown pixel that is between the corn leaf or another area that is not in the corn path? Which algorithm or methodology that I should study or apply in this problem?

EDIT1 : Using Spektre's answer and It looks better

Here is the result after I apply it with JAVA+Boofcv

• Step 1 : Thresholding or Color Segmentation

• Step 2 : Blured(use Gaussian and Median filter)

• Step 3 : Plot Linear Regression

More Information

Full Code Here

LinearRegression Class

10 example images with the same process

Answer 1

For your source image

I would:

1. create mask of brown stuff

   Just treshold H,S and ignore V, You already have this. I use integer 255 instead of color (Blue) for latter computation.

2. blur the mask

   This will remove small clusters of wrongly selected parts. After this you should treshold the mask again because the mask value will be a bit smaller then 255 unless you got fully selected areas. The bigger the area the bigger the value (closer to 255). I treshold with >=150

3. scan the mask by horizontal lines

4. for each line find the center of gravity of all selected pixels

   After blurring and tresholding again the used mask is in Aqua. So compute the average point x coordinate of all masked pixels in each line. This point is marked with White.

5. regress line through all of the centers of gravity

   I use mine approximation search for this but you can use any regression you want. The regressed line is marked with Red

   I used line equation x=x0+y*dx where y=<0,pic1.ys>. And search the solution on intervals:

   x0=<-pic1.xs,+2*pic1.xs>
   dx=<-10,+10>
   

Where pic1.xs,pic1.ys is the image resolution. So as you can see I do not cover full range of angles but I think this would not work on those edge cases anyway (near horizontal directions). For such cases you should do this on the Vertical lines instead and use x=y0+x*dy instead.

Here C++ source I did this with:

    picture pic0,pic1;
        // pic0 - source img
        // pic1 - output img
    int x,y,h,s,v,px,pn,*p;
    color c;
    // copy source image to output
    pic1=pic0;
    pic1.save("cornbot0.png");
    // create brown stuff mask
    for (y=0;y<pic1.ys;y++)             // scan all H lines
     for (x=0;x<pic1.xs;x++)            // scan actual H line
        {
        c=pic1.p[y][x];                 // get pixel color
        rgb2hsv(c);                     // in HSV
        h=WORD(c.db[picture::_h]);
        s=WORD(c.db[picture::_s]);
        v=WORD(c.db[picture::_v]);
        // Treshold brownish stuff
        if ((abs(h- 20)<10)&&(abs(s-200)<50)) c.dd=255; else c.dd=0;
        pic1.p[y][x]=c;
        }
    pic1.save("cornbot1.png");
    pic1.smooth(10);                    // blur a bit to remove small clusters as marked
    pic1.save("cornbot2.png");

    // compute centers of gravity
    p=new int[pic1.ys];                 // make space for points
    for (y=0;y<pic1.ys;y++)             // scan all H lines
        {
        px=0; pn=0;                     // init center of gravity (avg point) variables
        for (x=0;x<pic1.xs;x++)         // scan actual H line
         if (pic1.p[y][x].dd>=150)      // use marked points only
            {
            px+=x; pn++;                // add it to avg point
            pic1.p[y][x].dd=0x00004080; // mark used points (after smooth) with Aqua
            }
        if (pn)                         // finish avg point computation
            {
            px/=pn;
            pic1.p[y][px].dd=0x00FFFFFF;// mark it by White
            p[y]=px;                    // store result for line regression
            } else p[y]=-1;             // uncomputed value
        }

    // regress line
    approx x0,dx;
    double ee;
    for (x0.init(-pic1.xs,pic1.xs<<1,100,3,&ee); !x0.done; x0.step())   // search x0
     for (dx.init(-10.0   ,+10.0     ,1.0,3,&ee); !dx.done; dx.step())  // search dx
      for (ee=0.0,y=0;y<pic1.ys;y++)                                    // compute actua solution distance to dataset
       if (p[y]!=-1)                                                    // ignore uncomputed values (no brown stuff)
        ee+=fabs(double(p[y])-x0.a-(double(y)*dx.a));
    // render regressed line with Red
  for (y=0;y<pic1.ys;y++)
    {
    x=double(x0.aa+(double(y)*dx.aa));
    if ((x>=0)&&(x<pic1.xs))
     pic1.p[y][x].dd=0x00FF0000;
    }
    pic1.save("cornbot2.png");
    delete[] p;

I use my own picture class for images so some members are:

• xs,ys size of image in pixels
• p[y][x].dd is pixel at (x,y) position as 32 bit integer type
• p[y][x].dw[2] is pixel at (x,y) position as 2x16 bit integer type for 2D fields
• p[y][x].db[4] is pixel at (x,y) position as 4x8 bit integer type for easy channel access
• clear(color) - clears entire image
• resize(xs,ys) - resizes image to new resolution
• bmp - VCL encapsulated GDI Bitmap with Canvas access
• smooth(n) - fast blur the image n times

You can further improve this with segmentation (removing small clusters) based on area and position. Also you can ignore too big peaks in avg points between neighbors. Also you can detect sky and ignore whole area where sky is present.

[edit1] smooth

This is how my smooth looks like:

void picture::smooth(int n)
    {
    color   *q0,*q1;
    int     x,y,i,c0[4],c1[4],c2[4];
    bool _signed;
    if ((xs<2)||(ys<2)) return;
    for (;n>0;n--)
        {
        #define loop_beg for (y=0;y<ys-1;y++){ q0=p[y]; q1=p[y+1]; for (x=0;x<xs-1;x++) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x+1],pf); dec_color(c2,q1[x],pf);
        #define loop_end enc_color(c0,q0[x  ],pf); }}
        if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]);  } loop_end
        if (pf==_pf_s   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end
        if (pf==_pf_u   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end
        if (pf==_pf_ss  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end
        if (pf==_pf_uu  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end
        #undef loop_beg
        #define loop_beg for (y=ys-1;y>0;y--){ q0=p[y]; q1=p[y-1]; for (x=xs-1;x>0;x--) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x-1],pf); dec_color(c2,q1[x],pf);
        if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]);  } loop_end
        if (pf==_pf_s   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end
        if (pf==_pf_u   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end
        if (pf==_pf_ss  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end
        if (pf==_pf_uu  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end
        #undef loop_beg
        #undef loop_end
        }
    }

It simply weight average 3 pixels

(x,y)=(2*(x,y)+(x-1,y)+(x,y-1))/4

and after that do the same with

(x,y)=(2*(x,y)+(x+1,y)+(x,y+1))/4

to avoid shifting of image. Then this whole thing is looped n times and that is all. You can ignore the clamp and pixel-format options in this case it is pf==_pf_rgba but it uses only single channel anyway ... dec_color,enc_color just unpack,pack color channels into/from array of variables to avoid truncating and overflow issues on 8 bit channels and also to format/simplify the code a bit better (for different pixel format support)

btw smooth base is the same as convolution with

0.00 0.25 0.00
0.25 0.50 0.00
0.00 0.00 0.00

and

0.00 0.00 0.00
0.00 0.50 0.25
0.00 0.25 0.00

Answer 2

If I am right you're asking about the brown portions which are seen astray or in other portions of the background?

How did you get that last image? I assume you multiplied the original image by a mask? Even if you didnt, you can simply get the mask from the image by selecting wherever the image exists (any simple, very low threshold would do). (Apply adaptive thresholding ,an even more accurate version of the original to get a better mask)

Take that mask and clean it up using morphological operations, in your case closing would suffice. Morphology is made up of a plethora of operations which can give you an extremely clean image mask. Read up on them.