How to apply threshold within multiple rectangular bounding boxes in an image?

Question

My question is that: I have ROI's for the bounding boxes around the objects in an image. The ROI's are obtained by the Faster R-CNN. Now what I want is to apply the thresholding to get the object accurately contained within the bounding box. The ROI of this image was got by the Faster RCNN.

So, After getting the ROI's, I only selected ROI from the image and pasted on the black image of the same size and dimension which result in the following image.let say

As you can see that boxes are rectangular so in some places it covers some background area along with spikes. So, how can I apply thresholding to get only the spikes and other pixels turn to black?

EDIT: I've added the link to the ROI text file of the first image in the question

ROI file for first image

Answer 1

Color thresholding using cv2.inRange() should work here. I'm assuming you want to isolate the green area

Here's the main idea

• Convert image to HSV format since it is easier to represent color than RBG
• Perform color segmentation with a lower/upper threshold

You could also perform morphological operations to smooth or remove noise after obtaining the mask

---

import numpy as np
import cv2

image = cv2.imread('1.jpg')
result = image.copy()
image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([18, 0, 0])
upper = np.array([179, 255, 255])
mask = cv2.inRange(image, lower, upper)
result = cv2.bitwise_and(result,result, mask=mask)
cv2.imshow('result', result)
cv2.imwrite('result.png', result)
cv2.waitKey()

You can use a HSV color thresholder script to isolate the desired color range

import cv2
import sys
import numpy as np

def nothing(x):
    pass

# Create a window
cv2.namedWindow('image')

# create trackbars for color change
cv2.createTrackbar('HMin','image',0,179,nothing) # Hue is from 0-179 for Opencv
cv2.createTrackbar('SMin','image',0,255,nothing)
cv2.createTrackbar('VMin','image',0,255,nothing)
cv2.createTrackbar('HMax','image',0,179,nothing)
cv2.createTrackbar('SMax','image',0,255,nothing)
cv2.createTrackbar('VMax','image',0,255,nothing)

# Set default value for MAX HSV trackbars.
cv2.setTrackbarPos('HMax', 'image', 179)
cv2.setTrackbarPos('SMax', 'image', 255)
cv2.setTrackbarPos('VMax', 'image', 255)

# Initialize to check if HSV min/max value changes
hMin = sMin = vMin = hMax = sMax = vMax = 0
phMin = psMin = pvMin = phMax = psMax = pvMax = 0

img = cv2.imread('1.jpg')
output = img
waitTime = 33

while(1):

    # get current positions of all trackbars
    hMin = cv2.getTrackbarPos('HMin','image')
    sMin = cv2.getTrackbarPos('SMin','image')
    vMin = cv2.getTrackbarPos('VMin','image')

    hMax = cv2.getTrackbarPos('HMax','image')
    sMax = cv2.getTrackbarPos('SMax','image')
    vMax = cv2.getTrackbarPos('VMax','image')

    # Set minimum and max HSV values to display
    lower = np.array([hMin, sMin, vMin])
    upper = np.array([hMax, sMax, vMax])

    # Create HSV Image and threshold into a range.
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, lower, upper)
    output = cv2.bitwise_and(img,img, mask= mask)

    # Print if there is a change in HSV value
    if( (phMin != hMin) | (psMin != sMin) | (pvMin != vMin) | (phMax != hMax) | (psMax != sMax) | (pvMax != vMax) ):
        print("(hMin = %d , sMin = %d, vMin = %d), (hMax = %d , sMax = %d, vMax = %d)" % (hMin , sMin , vMin, hMax, sMax , vMax))
        phMin = hMin
        psMin = sMin
        pvMin = vMin
        phMax = hMax
        psMax = sMax
        pvMax = vMax

    # Display output image
    cv2.imshow('image',output)

    # Wait longer to prevent freeze for videos.
    if cv2.waitKey(waitTime) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()

Here's the result on the original image

Answer 2

In your TensorFlow detection, the output dictionary you get after you run the prediction has a field, "detection_scores".

output_dict = sess.run(tensor_dict,feed_dict={image_tensor: image})

Set a threshold on that,

 indexes=np.where(output_dict['detection_scores']>0.5)

Use the boxes, i.e. output_dict['detection_boxes'] only on those specific indexes which you filtered in the previous step.

[EDIT] Adding more code after the discussion in comments

#convert the image to hsv
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
#tune the numbers below accordingly
lower_green = np.array([60, 100, 50])
upper_green = np.array([60 , 255, 255])

mask = cv2.inRange(hsv, lower_green, upper_green)
res = cv2.bitwise_and(frame,frame, mask= mask)
#res has the output masked image

[EDIT] editing with the actual image given in the question

img=cv2.imread("idJyc.jpg")
lower_green = np.array([0, 10, 0])
upper_green = np.array([255 , 100, 255])
mask = cv2.inRange(img, lower_green, upper_green)
mask = np.abs(255-mask)
res = cv2.bitwise_and(img,img, mask=mask)
cv2.imshow("a",res)
cv2.waitKey(0)

Adding the output image for your reference.

Answer 3

If you re familiar with applying neural networks and you re having enough data. This task is perfectly suited for segmentation.

I recommend U-Net, since it works with a small amount of labelled data for training. It is also fast, with few operations for this task with comparably low complexity. And has shown good performance on various tasks.

I also found a full code pipeline, in this case for heart segmentation in zebrafish, but in my opinion they did a good job explaining how to prepare the data (proposing labeling tools, etc.) and train the model.

Also taking a step back, you could also think of interpreting your task as a segmentation task from the beginning. Especially for the U-Net it shouldn't be a problem to segment multiple instances in the same image.