Use a custom kernel / image filter to find a specific pattern in a 2d array

Question

Given an image im,

>>> np.random.seed(0)
>>> im = np.random.randint(0, 100, (10,5))
>>> im
array([[44, 47, 64, 67, 67],
       [ 9, 83, 21, 36, 87],
       [70, 88, 88, 12, 58],
       [65, 39, 87, 46, 88],
       [81, 37, 25, 77, 72],
       [ 9, 20, 80, 69, 79],
       [47, 64, 82, 99, 88],
       [49, 29, 19, 19, 14],
       [39, 32, 65,  9, 57],
       [32, 31, 74, 23, 35]])

what is the best way to find a specific segment of this image, for instance

>>> im[6:9, 2:5]
array([[82, 99, 88],
       [19, 19, 14],
       [65,  9, 57]])

If the specific combination does not exist (maybe due to noise), I would like to have a similarity measure, which searches for segments with a similar distribution and tells me for each pixel of im, how good the agreement is. For instance something like

array([[0.03726647, 0.14738364, 0.04331007, 0.02704363, 0.0648282 ],
       [0.02993497, 0.04446428, 0.0772978 , 0.1805197 , 0.08999   ],
       [0.12261269, 0.18046972, 0.01985607, 0.19396181, 0.13062801],
       [0.03418192, 0.07163043, 0.15013723, 0.12156613, 0.06500945],
       [0.00768509, 0.12685481, 0.19178985, 0.13055806, 0.12701177],
       [0.19905991, 0.11637007, 0.08287372, 0.0949395 , 0.12470202],
       [0.06760152, 0.13495046, 0.06344035, 0.1556691 , 0.18991421],
       [0.13250537, 0.00271433, 0.12456922, 0.97      , 0.194389  ],
       [0.17563869, 0.10192488, 0.01114294, 0.09023184, 0.00399753],
       [0.08834218, 0.19591735, 0.07188889, 0.09617871, 0.13773224]])

The example code is python. I think there should be a solution correlating a kernel with im. This will have the issue though, that a segment with the same value but scaled, will give a sharper response.

Answer 1

Template matching would be one of the ways to go about it. Of course deep learning/ML can also be used for more complicated matching.

Most image processing libraries support some sort of matching function which compares a set of 2 image - reference and the one to match. In OpenCV it returns a score which can used to determine a match. The matching method uses various functions that support scale and/or rotation invariant matching. Beware of licensing constraints in the method you plan to use.

In case the images may not always be exact, you can use standard deviation (StdDev) to allow for permissible deviation and yet classify them into buckets. Histogram matching may also be used depending on the condition of image to be matched (lighting, color can be important, unless you use specific channels). Use of histogram will avoid matching template in its entirety.

Ref for Template Matching:

• OpenCV - https://docs.opencv.org/master/d4/dc6/tutorial_py_template_matching.html
• SciPy - https://scikit-image.org/docs/dev/auto_examples/features_detection/plot_template.html

Answer 2

Thanks to banerjk for the great answer - template matching is exactly the solution!

some backup method

Considering my correlating-with-a-kernel idea, there is some progress: When one correlates the image with the template (i.e. what I called target segment in the question), chances are high, that the most intense point in the correlated image (relative to the mean intensity) matches the template position (see im and m in the example). Seems like I am not the first, who comes up with this idea, as can be see in these lecture notes on page 39.

However, this is not always true. This method, more or less, just detects weight at the largest values in the template. In the example, im2 is constructed such, that it tricks this concept.

Maybe it gets more reliable if one applies some filter (for instance median) on the image beforehand.

I just wanted to mention it here, as it might have advantages for certain situations (it should be more performant compared to the Wikipedia-implementation of template_matching).

example

import numpy as np
from scipy import ndimage

np.random.seed(0)
im = np.random.randint(0, 100, (10,5))
t = im[6:9, 2:5]
print('t', t, sep='\n')

m = ndimage.correlate(im, t) / ndimage.correlate(im, np.ones(t.shape))
m /= np.amax(m)

print('im', im, sep='\n')
print('m', m, sep='\n')

print("this can be 'tricked', however")

im2 = im.copy()
im2[6:9, :3] = 0
im2[6,1] = 1

m2 = ndimage.correlate(im2, t) / ndimage.correlate(im2, np.ones(t.shape))
m2 /= np.amax(m2)

print('im2', im2, sep='\n')
print('m2', m2, sep='\n')

output

t
[[82 99 88]
 [19 19 14]
 [65  9 57]]
im
[[44 47 64 67 67]
 [ 9 83 21 36 87]
 [70 88 88 12 58]
 [65 39 87 46 88]
 [81 37 25 77 72]
 [ 9 20 80 69 79]
 [47 64 82 99 88]
 [49 29 19 19 14]
 [39 32 65  9 57]
 [32 31 74 23 35]]
m
[[0.73776208 0.62161208 0.74504705 0.71202601 0.66743979]
 [0.70809611 0.70617161 0.70284942 0.80653741 0.67067733]
 [0.55047727 0.61675268 0.5937487  0.70579195 0.74351706]
 [0.7303857  0.77147963 0.74809273 0.59136392 0.61324214]
 [0.70041161 0.7717032  0.69220064 0.72463532 0.6957257 ]
 [0.89696894 0.69741108 0.64136612 0.64154719 0.68621613]
 [0.48509474 0.60700037 0.65812918 0.68441118 0.68835903]
 [0.73802038 0.83224745 0.87301124 1.         0.92272565]
 [0.72708573 0.64909142 0.54540817 0.60859883 0.52663327]
 [0.72061572 0.70357846 0.61626289 0.71932261 0.75028955]]
this can be 'tricked', however
im2
[[44 47 64 67 67]
 [ 9 83 21 36 87]
 [70 88 88 12 58]
 [65 39 87 46 88]
 [81 37 25 77 72]
 [ 9 20 80 69 79]
 [ 0  1  0 99 88]
 [ 0  0  0 19 14]
 [ 0  0  0  9 57]
 [32 31 74 23 35]]
m2
[[0.53981867 0.45483201 0.54514907 0.52098765 0.48836403]
 [0.51811216 0.51670401 0.51427317 0.59014141 0.49073293]
 [0.40278285 0.4512764  0.43444444 0.51642621 0.54402958]
 [0.5344214  0.56448972 0.54737758 0.43269951 0.44870774]
 [0.51248943 0.56465331 0.50648148 0.53021386 0.50906076]
 [0.78923691 0.56633529 0.51641414 0.44336403 0.50210263]
 [0.88137788 0.89779614 0.63552189 0.55070797 0.50367059]
 [0.88888889 1.         0.75544508 0.75694003 0.67515605]
 [0.43965976 0.48492221 0.37490287 0.48511085 0.38533625]
 [0.30754918 0.32478065 0.27066895 0.46685032 0.548985  ]]

Maybe someone can contribute on the background of the lecture notes.

update: It is discussed in J. P. Lewis, “Fast Normalized Cross-Correlation”, Industrial Light and Magic. on the very first page.