Find lines and color them with mouse click

Question

@ALL This is an edit of the original question to bring a little more light on the subject.

Problem Statement

• Suppose there is an industrial P&ID plot.
• Aiming to color only some lines important to the process.
• The user should only click (left mouse-click) on the line segment to get it colored.

Problem Approach

I am new to programming -> using Python (3.5) to try this out. The way I see it the algorithm is like this:

• The plot will be in .pdf format. Therefore I could employ PIL ImageGrab or convert .pdf to .png as presented in this example
• The algorithm will search for pixels around the mouse click, then compare it to another portion of identical size (let's say a strip of 6x3 px), but one step to the left/right (be it 1-5 px)
• Checking the mean of their differences will tell us if the two strips are identical
• This way the algorithm should find both the line endings, arrows, corners or other elements
• Once this is found, the positions recorded and the markup line drawn, the user is expected to pick another line

Summed up

• Click on the wanted line
• Grab a small portion of the image around the mouse click
• Check if the line is either horizontal or vertical
• Crop an horizontal/vertical slice of a given size
• Find line endings and record the endings positions
• Between the two found positions draw a line of a certain color (let's say green)
• Wait for the next line to be picked and repeat

Other thoughts

• Attached you can find two pictures of a sample image and what I am trying to achieve.
• Tried to find "holes" in the slices using the approach found here: OpenCV to find line endings
• There is no strict rule in sticking with ImageGrab routine or anything alike
• If you know other tactics that I could use, please feel free to comment
• Any advice is welcome and sincerely appreciated

Sample image:

Desired Result (modified in Paint):

Adding an update to the post with the work I tried out so far

I've done some modifications on the original code so I will post it below. Everything in comments is either for debug or explanations. Your help is highly appreciated! Do not be afraid to intervene.

import win32gui as w
from PIL import ImageStat, ImageChops, Image, ImageDraw
import win32api as wa

img=Image.open("Trials.jpg")
img_width=img.size[0]
img_height=img.size[1]
#Using 1920 x 1080 resolution
#Hide the taskbar to center the Photo Viewer
#Defining a way to make sure the mouse click is inside the image
#Substract the width from total and divide by 2 to get base point of the crop
width_lim = (1920 - img_width)/2
height_lim = (1080 - img_height)/2-7
#After several tests, the math in calculating the height is off by 7 pixels, hence the correction
#Use these values when doing the crop

#Check if left mouse button was pressed and record its position
left_p = wa.GetKeyState(0x01)
#print(left_p)
while True :
    a=wa.GetKeyState(0x01)
    if a != left_p:
        left_p = a
        if a<0 :
            pos = w.GetCursorPos()
            pos_x=pos[0]-width_lim
            pos_y=pos[1]-height_lim
#            print(pos_x,pos_y)
        else:
            break


#img.show()
#print(img.size)

#Define the crop height; size is doubled
height_size = 10
#Define max length limit
#Getting a horizontal strip
im_hor = img.crop(box=[0, pos_y-height_size, img_width, pos_y+height_size])
#im_hor.show()



#failed in trying crop a small square of 3x3 size using the pos_x
#sq_size = 3
#st_sq = im_hor.crop(box=[pos_x,0,pos_x+sq_size,height_size*2])
#st_sq.show()

#going back to the code it works
#crop a standard strip and compare with a new test one
#if the mean of difference is zero, the strips are identical
#still looking for a way to find the position of the central pixel (that would be the one with maximum value - black)
strip_len = 3
step = 3
i = pos_x
st_sq = im_hor.crop(box=[i,0,i+strip_len,height_size*2])
test_sq = im_hor.crop(box=[i+step,0,i+strip_len+step,height_size*2])
diff = ImageChops.difference(st_sq,test_sq)
stat=ImageStat.Stat(diff)
mean = stat.mean
mean1 = stat.mean
#print(mean)

#iterate to the right until finding a different strip, record position
while mean==[0,0,0]:
    i = i+1
    st_sq = im_hor.crop(box=[i,0,i+strip_len,height_size*2])
    #st_sq.show()
    test_sq = im_hor.crop(box=[i+step,0,i+strip_len+step,height_size*2])
    #test_sq.show()
    diff = ImageChops.difference(st_sq,test_sq)
    #diff.show()
    stat=ImageStat.Stat(diff)
    mean = stat.mean
#    print(mean)
print(i-1)

r = i-1
#print("STOP")
#print(r)
#record the right end as r = i-1

#iterate to the left until finding a different strip. record the position
while mean1==[0,0,0]:
    i = i-1
    st_sq = im_hor.crop(box=[i,0,i+strip_len,height_size*2])
    #st_sq.show()
    test_sq = im_hor.crop(box=[i+step,0,i+strip_len+step,height_size*2])
    #test_sq.show()
    diff = ImageChops.difference(st_sq,test_sq)
    #diff.show()
    stat=ImageStat.Stat(diff)
    mean1 = stat.mean
#    print(mean)
#print("STOP")
print(i+1)

l = i+1
#record the left end as l=i+1
test_draw = ImageDraw.Draw(img)
test_draw.line([l,pos_y,r,pos_y], fill=128)
img.show()

#find another approach or die trying!!! 

Below is the result I got. It is not what I was hoping for, but I feel like being on the right track. I could really use some help on finding the pixel position in a strip and make it relative to the big picture pixel position.

Another image of the sort, in better quality, but yet bringing more problems into the fray.

Answer 1

So this solution is not a full solution to your exact problem, but I think it might be a good approach that can get you at least part of the way there. The issues I have in general with line detection approaches is that they usually heavily rely on multiple hyperparameters. More annoyingly, they are slow since they are searching a wide array of angles; your lines are strictly either horizontal or vertical. As such, I'd recommend using morphology. You can find a general overview of morphology on the OpenCV site and you can see it applied to remove the music bars in a score on this tutorial on the OpenCV site.

The basic idea I thought was:

1. Detect horizontal and vertical lines
2. Run connectedComponents() on the detected lines to identify each line separately
3. Get user mouse position and define a window around it
4. If a label from the connected components is in that window, then grab that component
5. Draw that component on the image

Now, this is a very basic idea and ignores some of the challenges involved. However, what this will do for sure is, if you click anywhere and there is a line in your image within the window of that click, you will get it. There are no missed lines here. Additional good news is it doesn't ignore the thicker borders and such in the image where you would naturally want this to stop (note this problem exists for line detection schemes). This will only detect lines that have a defined width and if the line gets thicker (turns into an arrow or hits a line going a different direction) it cuts it off. The bad news is that this uses pre-defined width for your lines. You can somewhat skirt around this by using a hit-or-miss transform, but note that the implementation is currently broken for versions of OpenCV older than 3.3-rc; see here for more (you can get around the broken implementation easily). Anyways, a hit-or-miss transform here allows you to say "I want a horizontal line but it can be a few pixels wide or just one pixel wide". Of course the wider you make it, the more things that aren't lines might turn into one. You can filter these out later though based on the size (toss all lines that are smaller than some size with erosion or dilation).

---

Now what does that look like in code? I decided to make a simple example and apply this, but note the code is thrown together so there's no real error catching here, and you'd want to write this a lot nicer. Either way it's just a quick hack to give an example of the above method.

First, we'll create the image and draw some lines:

import cv2
import numpy as np 

img = 255*np.ones((500, 500), dtype=np.uint8)
cv2.line(img, (10, 350), (200, 350), color=0, thickness=1)
cv2.line(img, (100, 150), (400, 150), color=0, thickness=1)
cv2.line(img, (300, 250), (300, 500), color=0, thickness=1)
cv2.line(img, (100, 50), (100, 350), color=0, thickness=1)
bin_img = cv2.bitwise_not(img)

And note I've also created the opposite image, because I prefer to keep the things I'm trying to detect white with black being the background.

Now we'll grab those horizontal and vertical lines with morphology (erosion in this case):

h_kernel = np.array([[0, 0, 0],
                     [1, 1, 1],
                     [0, 0, 0]], dtype=np.uint8)
v_kernel = np.array([[0, 1, 0],
                     [0, 1, 0],
                     [0, 1, 0]], dtype=np.uint8)

h_lines = cv2.morphologyEx(bin_img, cv2.MORPH_ERODE, h_kernel)
v_lines = cv2.morphologyEx(bin_img, cv2.MORPH_ERODE, v_kernel)

And now we'll label each line:

h_n, h_labels = cv2.connectedComponents(h_lines)
v_n, v_labels = cv2.connectedComponents(v_lines)

These images h_labels and v_labels will be identical to h_lines and v_lines but instead of the color/value being white at each pixel, the value is instead an integer for each different component in the image. So the background pixels will have the value 0, one line will be labeled with 1s, the other labeled with 2s. And so on for images with more lines.

Now we'll define a window around a user mouse-click. Instead of implementing that pipeline here I'm just going to hard-code a mouse-click position:

mouse_click = [101, 148]  # x, y
click_radius = 3  # pixel width around mouse click
window = [[mouse_click[0] - i, mouse_click[1] - j]
          for i in range(-click_radius, click_radius+1)
          for j in range(-click_radius, click_radius+1)]

The last thing to do is loop through all the locations inside the window and check if the label there is positive (i.e. it's not background). If it is, then we've hit a line. So now we can just look at all the pixels that have that label, and that will be the full line. Then we can use any number of methods to draw the line on the original img.

label = 0
for pixel in window:
    if h_labels[pixel[1], pixel[0]] > 0:
        label = h_labels[pixel[1], pixel[0]]
        bin_labeled = 255*(h_labels == label).astype(np.uint8)
    elif v_labels[pixel[1], pixel[0]] > 0:
        label = v_labels[pixel[1], pixel[0]]
        bin_labeled = 255*(v_labels == label).astype(np.uint8)
    if label > 0:
        rgb_labeled = cv2.merge([img, img+bin_labeled, img])
        break

IMO this code directly above is really sloppy, there are better ways to draw this but I didn't want to spend time on something not really central to the question.

---

One easy way to improve this would be to connect near lines---you could do this still with morphology, before you find the components. A better way to draw would probably be to simply find the min/max locations of that label inside the image, and just use those as the endpoint coordinates for OpenCV's line() function to draw, which would allow you to choose colors and line thicknesses easily. One thing I'd suggest doing if possible would be to show these lines on a mouseover before the user clicks (so they know they're clicking in the right area). That way if a user is close to two lines they know which one they're selecting.