How to separate the color fills from main big box and save them with correct labels

Question

Here is the main input image (named 1.png):

Now, I want to identify which box is filled with color and which box is empty, in the center big box and not the 20 side boxes that have number.

I wrote the code to extract main big box from image:

image = cv2.imread(path)

gray = cv2.cvtColor(image , cv2.COLOR_BGR2GRAY)

edge = cv2.Canny(gray.copy() , 10 ,70)

_,contours,_ = cv2.findContours(edge.copy() , cv2.RETR_EXTERNAL ,cv2.CHAIN_APPROX_SIMPLE)

cv2.drawContours(image, contours , -1, (0,0,255) , 3)
plt.imshow(image)
cv2.imwrite('image.jpg',image)

Now the image looks like this:

Then I sorted the contours to get main matrix box which has all the colored and non colored boxes.

sorted_contours = sorted(contours,key = cv2.contourArea,reverse=True)

Then I seperated the main box:

img = cv2.imread('1.png')
cnt = sorted_contours[0]
x,y,w,h = cv2.boundingRect(cnt)
main_box = img[y:y+h,x:x+w].copy()
cv2.imwrite('main_box.jpg',main_box)
plt.imshow(main_box)

So, the main box looks like this now:

Mark the exterior box

main_box_gray = cv2.cvtColor(main_box,cv2.COLOR_BGR2GRAY)
_, main_box_gray = cv2.threshold(main_box_gray,5,255,cv2.THRESH_BINARY)


_,t_c,_ = cv2.findContours(main_box_gray.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)

colored_main_box = img[y:y+h,x:x+w].copy()

cv2.drawContours(colored_main_box,t_c,-1,(255,0,0),2)

The outer box is marked,now I separated the outer contour to get boxes inside:

sorted_box = sorted(t_c,key = cv2.contourArea,reverse=True)

colored_main_box = img[y:y+h,x:x+w].copy()
cnt = sorted_box[0]

x2,y2,w2,h2 = cv2.boundingRect(cnt)

temp_image = colored_main_box[y2:y2+h2,x2:x2+w2].copy()


edge_temp = cv2.Canny(temp_image,100,200)

_,t_c_1,_ = cv2.findContours(edge_temp.copy(),cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)

colored_main_box = img[y:y+h,x:x+w].copy()

colored_main_box = colored_main_box[y2:y2+h2,x2:x2+w2].copy()

cv2.drawContours(colored_main_box,t_c_1,-1,(0,0,255),1)

plt.imshow(colored_main_box)
cv2.imwrite("full_marked.jpg",colored_main_box)

colored_main_box (Full marked):

Now I have marked all inner boxes and i can find which box are filled by using cv2.countNonZero funtion, but my main task is to save color inside the filled box as a separate image without the exterior black border of the box that contains that color fill, and the saved image that contain only color fill, should be named after the box that it was extracted.

e.g.: in main_box.jpg image:

The orange color should be saved as 8.jpg.

The Red color should be saved as 12.jpg.

The green color should be saved as 18.jpg.

The yellow should be saved as 19.jpg.

The blue color should be number 21.jpg.

Please help me with the code to extract and save the color fills inside the boxes with correct number label.

Answer 1

there are two subset of problems that you need to solve.

1. getting Bounding box for colored boxes.
2. naming them using box indices So first we need to crop image corresponding to each color. To get the colors in Image use this:

unique_col = set( tuple(v) for col in image for v in col )     
for color in unique_col:
    if col == (0,0,0) or col == (255,255,255):
       continue
    idx_set = np.where((image == col).all(axis=2))
    min_x = min(idx_set[:,0])
    min_y = min(idx_set[:,1])
    max_x = max(idx_set[:,0])
    max_y = max(idx_set[:,1]) 
    #this will give you bounding box for that particular color

step 2: assigning correct names let bbox_small be bounding box of each small boxes in the main bigger box then sort it according to y coordinate of just one corner of all the rectangle(=sorted_bbox array). and assign key(id) to each box = index of that box in sorted_bbox + 1

now

for every colored box:
    name = i+1 such that colored_box lies inside sorted_box[i]

I have not written any of the code, its just an idea. Also I think using Hough Transform to get corners of rectangle is also a good approach.