4

I am facing a problem of image clustering based on their similarity, without knowing the number of clusters. Ideally i would like to achieve something that resembles this http://cs231n.github.io/assets/cnnvis/tsne.jpeg (http://cs231n.github.io/understanding-cnn/ this picture is a result of convolution neural network and it represents groups it learnt)

Because I am not interested in their classification (I don't know classes), i am mostly interested in their 'visual' properties: colours, shapes, gradients etc. I have found number of articles suggesting algorithms like DBSCAN, t-SNE or even k-means but is there some better solution? There were suggestions of using HOG transformation but to be honest, no idea how to stitch it all together.

So, to summaries, how can I segregate (on 2D plane, into groups, folders, whatever) images based on their colours and shape properties?

Bartek Wójcik
  • 441
  • 2
  • 13
  • 1
    That image does *not* show a clustering, but a visualization. Make sure you have understood the different steps performed there (or not performed - no clustering). The entire topic is too complex to be just answered here - CNNs are a complex topic. Too broad to be answered / tutorial request -> voting to close as off-topic. – Has QUIT--Anony-Mousse Oct 20 '19 at 07:48
  • you are right, i needed visualisation rather than clustering. Thanks – Bartek Wójcik Oct 20 '19 at 09:27

2 Answers2

4

t-SNE is actually perfect for the thing you are trying to do.

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a (prize-winning) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.

You can read more about it here.

As always sklearn has a very user-friendly TSNE object to quickly try it out.

I hope this helps...

Sharif Elfouly
  • 378
  • 3
  • 9
1

I probably understand why people down-vote this question: because there are hardly any good answers from years of related posts, and the problem is painfully complex. More to the point, though:

Unfortunately image information dimensionality is higher than 2D. The photo you link is just a projection from high-dimensional space to a plane, and not necessarily representative of how the actual space looks like. This specific projection is mostly about colors, isn't it?

The same applies to trying to squeeze clusters into folders - in most cases it is impossible.

Yet you are correctly pointing to specific comparison dimensions in your question: how can I segregate (on 2D plane, into groups, folders, whatever) images based on their colors and shape properties?

The solution is to focus on the similarity dimension/metric of your interest. E.g. specifically "does this image contain a circle?", and optimize for this. But if you want a "square", you are already in another dimension. If optimizing for color, you can look at "overall redness". The more metrics you add, the higher is your clustering dimensionality.

Our perception is like this. We aim at specific summary metric, e.g. a scalar value, which is a sum of weighted metrics in different dimensions (ranking problem). For example, if you want photos with "eyes", you do not care about color variations. And vice versa, if you care more about colors, shapes are less important.

From my experience, clustering is easier when image clusters are relatively uniform. E.g. pictures in each potential cluster are very similar by some metric. For example one group is "bridges", another "faces". If you have very diverse images of any possible subject (even noise), the solution is intractable, unless you specify what exactly you want to group by.

Similar.Pictures
  • 217
  • 1
  • 9