I have noticed that tf.nn.softmax_cross_entropy_with_logits asks for "unscaled log probabilities" for the logits argument. However, nowhere have I seen anyone suggest performing a log operation on their NN predictions before submission to this function. Am I missing something here?
Asked
Active
Viewed 1,801 times
3
SuaveSouris
- 1,060
- 15
- 17
-
See this previous posting. http://stackoverflow.com/questions/34240703/difference-between-tensorflow-tf-nn-softmax-and-tf-nn-softmax-cross-entropy-with. – koshy george Nov 29 '16 at 18:26
-
I have but it doesn't address my particular question – SuaveSouris Nov 29 '16 at 18:42
2 Answers
7
You shouldn't perform a log operation. You shouldn't perform anything, actually :-). What this comment is (arguably poorly) trying to say is that each logit is an unrestricted real number (negative or positive, as big or as small as you want). The softmax cross entropy function will then (conceptually) apply the softmax operation (exponentiate, to turn unrestricted real numbers into positive numbers, and then normalize to make them sum to 1) and compute the cross-entropy.
So, tl;dr, feed the outputs of your last linear layer without any normalization or transfer function to this and you won't be wrong.
Alexandre Passos
- 4,943
- 1
- 11
- 19
0
for historical reasons, the output of last linear layer say:
Z2=np.dot(W2,A1)+b2 ,
this output is called "unscaled log probabilities" thus the softmax operation will normalize them --> giving the "scaled log probabilities" and apply the exponential function --> giving the "scaled probabilities" --> on which it applies the cross-entropy loss
Hatem Zegnani
- 1
- 1