12

I understand conceptually what is happening in a max/sum pool as a CNN layer operation, but I see this term "max pool over time", or "sum pool over time" thrown around (e.g., "Convolutional Neural Networks for Sentence Classification" paper by Yoon Kim). What is the difference?

Maxim
  • 47,916
  • 23
  • 132
  • 189
Matt
  • 1,159
  • 3
  • 16
  • 29

2 Answers2

14

The max-over-time pooling is usually applied in NLP (unlike ordinary max-pool, which is common in CNNs for computer vision tasks), so the setup is a little bit different.

The input to the max-over-time pooling is a feature map c = [c(1), ..., c(n-h+1)], which is computed over a sentence of length n with a filter of size h. The convolution operation is very similar to one with images, but in this case it's applied to 1-dimensional vector of words. This is the formula (3) in the paper.

The max-over-time pooling operation is very simple: max_c = max(c), i.e., it's a single number that gets a max over the whole feature map. The reason to do this, instead of "down-sampling" the sentence like in a CNN, is that in NLP the sentences naturally have different length in a corpus. This makes the feature maps different for different sentences, but we'd like to reduce the tensor to a fixed size to apply softmax or regression head in the end. As stated in the paper, it allows to capture the most important feature, one with the highest value for each feature map.

Note that in computer vision, images are usually1 of the same size, like 28x28 or 32x32, that's why it is unnecessary to downsample the feature maps to 1x1 immediately.

Sum-pooling-over-time is the same.

1 Modern CNN can be trained with images of different size, but this requires the network to be all-convolutional, so it doesn't have any pooling layers. See this question for more details.

Maxim
  • 47,916
  • 23
  • 132
  • 189
  • Regarding your footnote: I believe Global Average Pooling supports input images of different sizes. – Imran Jan 31 '18 at 19:39
  • Correct. Good comment. But it's much closer to 1x1 convolution, than to avg-pool layer, so the network still won't have ordinary pooling layers. – Maxim Jan 31 '18 at 19:42
  • seriously?! the way that they force a fixed size layer is by taking the maximum and getting a SINGLE number as a layer? That seems crazy to me. Then how do they even apply a softmax layer at the end? – Charlie Parker Apr 25 '19 at 22:42
5

Max pooling typically applies to regions in a 2d feature plane, while max pooling over time happens along a 1d feature vector.

Here is a demonstration of max pooling from Stanford's CS231n:

max pooling

Max pooling over time takes a 1d feature vector and computes the max. The "over time" just means this is happening along the time dimension for some sequential input, like a sentence, or a concatenation of all phrases from a sentence as in the paper you linked.

For example:

[2, 7, 4, 1, 5] -> [7]

Source: CS224d Lecture 13 slides

Imran
  • 9,796
  • 7
  • 55
  • 71
  • 1
    While this is absolutely correct, this is not relevant to the NLP related question above, but more for computer vision. – cmcapellan Feb 16 '21 at 00:07